Person Detection, Person Identification and Meeting Start for Interactive Whiteboard Appliances

ABSTRACT

Artificial intelligence is introduced into an electronic meeting context to perform various tasks before, during, and/or after electronic meetings. The artificial intelligence may analyze a wide variety of data such as data pertaining to other electronic meetings, data pertaining to organizations and users, and other general information pertaining to any topic. Capability is also provided to create, manage, and enforce meeting rules templates that specify requirements and constraints for various aspects of electronic meetings. Embodiments include improved approaches for translation and transcription using multiple translation/transcription services. Embodiments also include using sensors in conjunction with interactive whiteboard appliances to perform person detection, person identification, attendance tracking, and improved meeting start. Embodiments further include improvements to the presentation of content on interactive whiteboard appliances, providing meeting services for meeting attendees, agenda extraction, and learning to aid in creating new electronic meetings.

RELATED APPLICATION DATA

This application is related to U.S. patent application Ser. No.14/992,273 (Attorney Docket No. 49986-0859) entitled “Electronic MeetingIntelligence”, filed Jan. 11, 2016, and Ser. No. 14/992,278 (AttorneyDocket No. 49986-0860) entitled “Electronic Meeting Intelligence”, filedJan. 11, 2016, and U.S. patent application Ser. No. 15/384,181 (AttorneyDocket No. 49986-0883) entitled “Approach For Accessing Third-PartyContent Collaboration Services on Interactive Whiteboard AppliancesUsing A Wallpaper”, filed Dec. 19, 2016, and U.S. patent applicationSer. No. 15/384,184 (Attorney Docket No. 49986-0884) entitled “ApproachFor Accessing Third-Party Content Collaboration Services on InteractiveWhiteboard Appliances Using Cross-License Authentication”, filed Dec.19, 2016, and U.S. patent application Ser. No. 15/384,187 (AttorneyDocket No. 49986-0885) entitled “Approach For Accessing Third-PartyContent Collaboration Services on Interactive Whiteboard Appliances ByAn Application Using a Wrapper Application Program Interface”, filedDec. 19, 2016, and U.S. patent application Ser. No. 15/290,855 (AttorneyDocket No. 49986-0888) entitled “Managing Electronic Meetings UsingArtificial Intelligence and Meeting Rules Templates”, filed Ser. No.15/290,856, and U.S. patent application Ser. No. 15/290,856 (AttorneyDocket No. 49986-0889) entitled “Creating Agendas for ElectronicMeetings Using Artificial Intelligence”, filed Oct. 11, 2016, and U.S.patent application Ser. No. 15/290,858 (Attorney Docket No. 49986-0890)entitled “Selecting Meeting Participants for Electronic Meetings UsingArtificial Intelligence”, filed Oct. 11, 2016, and U.S. patentapplication Ser. No. 15/290,860 (Attorney Docket No. 49986-0891)entitled “Real-Time (Intra-Meeting) Processing Using ArtificialIntelligence”, filed Oct. 11, 2016, and U.S. patent application Ser. No.15/290,861 (Attorney Docket No. 49986-0892) entitled “Post-MeetingProcessing Using Artificial Intelligence”, filed Oct. 11, 2016, and U.S.patent application Ser. No. 15/477,240 (Attorney Docket No. 49986-0902)entitled “Electronic Meeting Intelligence”, filed Apr. 3, 2017, and U.S.patent application Ser. No. 15/477,276 (Attorney Docket No. 49986-0903)entitled “Electronic Meeting Intelligence”, filed Apr. 3, 2017, and U.S.patent application Ser. No. ______ (Attorney Docket No. 49986-0907)entitled “Speech-to-Text Conversion for Interactive WhiteboardAppliances Using Multiple Services”, filed Oct. 6, 2017, and U.S. patentapplication Ser. No. ______ (Attorney Docket No. 49986-0910) entitled“Speech-to-Text Conversion for Interactive Whiteboard Appliances inMulti-Language Electronic Meetings”, filed Oct. 6, 2017, and U.S. patentapplication Ser. No. ______ (Attorney Docket No. 49986-0911) entitled“Attendance Tracking, Presentation Files, Meeting Services and AgendaExtraction for Interactive Whiteboard Appliances”, filed Oct. 6, 2017,and U.S. patent application Ser. No. ______ (Attorney Docket No.49986-0912) entitled “Interactive Whiteboard Appliances With LearningCapabilities”, filed Oct. 6, 2017, the contents all of which areincorporated by reference in their entirety for all purposes as if fullyset forth herein.

FIELD

Embodiments relate to interactive whiteboard appliances (IWBs) forconducting electronic meetings. SUGGESTED GROUP ART UNIT: 2653;SUGGESTED CLASSIFICATION: 348.

BACKGROUND

The approaches described in this section are approaches that could bepursued, but not necessarily approaches that have been previouslyconceived or pursued. Therefore, unless otherwise indicated, it shouldnot be assumed that any of the approaches described in this sectionqualify as prior art merely by virtue of their inclusion in thissection.

A meeting is typically an effective vehicle for coordinating thesuccessful accomplishment of a common goal shared by multiple people.However, a meeting can also devolve into a counterproductive use of timein the absence of proper organization of the meeting itself. Forexample, too much time may be devoted to a particular topic thatinvolves a small subset of meeting attendees, and this may result inwasted time for the remaining attendees. Such circumstances may beavoided through the use of a person serving as a meeting moderator, butpersonal biases may affect the neutrality of the person serving as themeeting moderator. Such circumstances may also be avoided throughadequate preparation for the meeting, but it may be impossible toforesee all the possible issues that may arise during the meeting.

Another way for a meeting to result in wasted time is by failing tofully reap the benefits provided by the meeting. For example,transcribing the meeting, scheduling an additional meeting, analyzingmeeting participation, and/or researching an issue that was contendedduring the meeting may be tedious follow-up actions that are neglectedafter the meeting. Even if the follow-up actions are performed, theprocess of performing them may be slow and cost-prohibitive.

Thus, it is desirable and beneficial to perform the administrativeduties related to a meeting using an approach without the aforementionedshortcomings.

SUMMARY

According to an embodiment, an apparatus comprises one or moreprocessors and one or more memories storing instructions. Processing ofthe instructions by one or more processors causes receiving, over one ormore communications networks from a requestor device, audio data thatrepresents human speech or text. Two or more translation/transcriptionservices are selected from a plurality of translation/transcriptionservices to process the audio data that represents human speech or text.The audio data that represents human speech or text is provided to thetwo or more selected translation/transcription services andtranslation/transcription data is received from eachtranslation/transcription service that includes a plurality of units ofspeech and a plurality of confidence scores for the plurality of unitsof speech. Resulting translation/transcription data that includes aplurality of units of speech selected from the translation/transcriptiondata received from the two or more selected translation/transcriptionservices is generated based upon one or more selection criteria thatinclude confidence scores. The resulting translation/transcription datais provided over the one or more communications networks to therequestor device.

According to another embodiment, an apparatus comprises one or moreprocessors and one or more memories storing instructions. Processing ofthe instructions by one or more processors causes receiving, over one ormore communications networks from a requestor device, a request togenerate a transcript of audio data for a meeting. In response toreceiving, over one or more communications networks from a requestordevice, the request to generate a transcript of audio data, the audiodata is processed and a meeting transcript is generated for the meetingin a first language, along with a first command transcript for themeeting in the first language, and a second command transcript for themeeting in a second language that is different than the first language.The meeting transcript for the meeting in the first language, the firstcommand transcript for the meeting in the first language, and the secondcommand transcript for the meeting in the second language that isdifferent than the first language, are provided over the one or morecommunications networks to the requestor device.

According to yet another embodiment, an interactive whiteboard appliance(IWB) comprises a display, a communications interface, one or moreprocessors, and one or more memories storing instructions. Processing ofthe instructions by one or more processors causes receiving, via thecommunications interface, from one or more sensors, sensor data thatindicates a detected presence of a person. Based upon the sensor datathat indicates a detected presence of a person and identity informationthat specifies one or more of image data or voice data for each of aplurality of persons, a particular person is identified from theplurality of persons that corresponds to the sensor data that indicatesa detected presence of a person. A determination is made, based uponmeeting information and a current time, whether an electronic meetinginvolving the IWB appliance is scheduled at or around the current time.In response to determining, based upon the meeting information and thecurrent time, that an electronic meeting involving the IWB appliance isscheduled at or around the current time, a determination is made, basedupon the meeting information, whether the particular person is ascheduled participant of the particular electronic meeting involving theIWB appliance that is scheduled at or around the current time. Inresponse to determining, based upon the meeting information, that theparticular person is a scheduled participant of the particularelectronic meeting involving the IWB appliance that is scheduled at oraround the current time, one or more notifications pertaining to theparticular electronic meeting are generated and displayed on the displayof the IWB appliance.

According to a further embodiment, an interactive whiteboard appliance(IWB) comprises a display, a communications interface, one or moreprocessors, and one or more memories storing instructions. Processing ofthe instructions by one or more processors causes receiving, via thecommunications interface, from one or more sensors, first sensor datathat indicates a detected presence of a person within a specifieddistance of the IWB appliance. Based upon the first sensor data thatindicates a detected presence of a person within the specified distanceof the IWB appliance and identity information that specifies identifyinformation for a plurality of persons, a particular person isidentified from the plurality of persons that corresponds to the sensordata that indicates a detected presence of a person within the specifieddistance of the IWB appliance. A determination is made, based uponmeeting information and a current time, whether an electronic meetinginvolving the IWB appliance is scheduled at or around the current time.In response to determining, based upon the meeting information and thecurrent time, that an electronic meeting involving the IWB appliance isscheduled at or around the current time, a determination is made, basedupon the meeting information, whether the particular person is ascheduled participant of the particular electronic meeting involving theIWB appliance that is scheduled at or around the current time. Inresponse to determining, based upon the meeting information, that theparticular person is a scheduled participant of the particularelectronic meeting involving the IWB appliance that is scheduled at oraround the current time, first participation time data is generated andstored that specifies a time at which the particular person joined theparticular electronic meeting. Second sensor data that indicates thatthe particular person is no longer within the specified distance of theIWB appliance is received, via the communications interface, from theone or more sensors. In response to receiving, via the communicationsinterface, from the one or more sensors, the second sensor data thatindicates that the particular person is no longer within the specifieddistance of the IWB appliance, second participation time data thatspecifies a time at which the particular person is no longer aparticipant of the particular electronic meeting is generated andstored.

According to yet a further embodiment, an interactive whiteboardappliance (IWB) comprises a display, one or more processors, and one ormore memories storing instructions. Processing of the instructions byone or more processors causes retrieving IWB appliance usage informationthat specifies a plurality of IWB appliance usage values that were usedfor two or more electronic meetings that were previously conducted usingthe IWB appliance. The IWB appliance usage information is analyzed toidentify one or more usage patterns, wherein each usage pattern includesat least an IWB appliance usage value that was used in the two or moreelectronic meetings that were previously conducted using the IWBappliance. During the creation of a new electronic meeting involving theIWB appliance, including, in the new electronic meeting involving theIWB appliance, from the IWB usage patterns, at least one IWB applianceusage value that was used in the two or more electronic meetings thatwere previously conducted using the IWB appliance.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings:

FIG. 1A-C depict example computer architectures upon which embodimentsmay be implemented.

FIG. 2A depicts an example electronic meeting screen displayed by anelectronic meeting application at a node.

FIG. 2B depicts an example meeting rules templates screen displayed inresponse to a user selecting the “Meeting Rules Templates” option fromoptions in FIG. 2A.

FIG. 2C depicts a meeting rules templates screen that displays thecontents of the “Board of Directors” meeting rules template selected bya user from the meeting rules templates screen in FIG. 2B.

FIG. 2D depicts an electronic meeting screen displayed by an electronicmeeting application in response to a user selecting the “ElectronicMeetings” option from options in FIG. 2A.

FIG. 2E depicts an electronic meeting management screen displayed by anelectronic meeting application in response to a user selecting a controlfrom meeting controls.

FIG. 2F depicts a view electronic meeting screen displayed in responseto a user selecting the “View” option for the Code Review Meeting forthe Pluto Project displayed in existing meetings portion.

FIG. 2G depicts an electronic meeting management screen with a pop-upwindow that provides suggestions for missing meeting information, whichin the present example is the missing meeting owner.

FIG. 2H depicts an agenda creation screen generated in response to auser selecting Meeting Agenda control from electronic meeting managementscreen.

FIG. 2I depicts suggested agenda items for each category of suggestedagenda items depicted in FIG. 2H.

FIG. 2J depicts a participant selection screen generated in response toa user selecting Meeting Participants control from electronic meetingmanagement screen.

FIG. 2K depicts a natural language query via query window of anelectronic meeting screen that allows a user to enter a natural languagequery.

FIG. 3 is a block diagram that depicts an arrangement for generatingintervention data.

FIG. 4A is a block diagram that depicts an arrangement in which meetingintelligence apparatus includes speech or text recognition logic thatprocesses first meeting content data to determine one or morecorresponding agenda topics.

FIG. 4B is a block diagram that depicts an arrangement for performingsentiment analysis with respect to an ongoing discussion.

FIG. 4C is a block diagram that depicts an arrangement for retrievingrequested information using natural language queries.

FIG. 4D is a block diagram that depicts an arrangement for supplementingmeeting content with participant identification data.

FIG. 5 is a block diagram that depicts an arrangement for generating areport.

FIG. 6A is a block diagram that depicts an arrangement for generatingmeeting content metadata that includes participant identification data.

FIG. 6B is a block diagram that depicts an arrangement for generatingmeeting content metadata that includes a sentiment detected in firstmeeting content data.

FIG. 6C is a block diagram that depicts an arrangement for generatingmeeting content metadata that includes a label to identify a key, i.e.,important, meeting point.

FIG. 7A depicts an example meeting reports screen that provides accessto meeting results data for a plurality of electronic meetings.

FIG. 7B depicts an example meeting results screen for the Comet DesignReview Meeting.

FIG. 7C depicts an example participant analysis report.

FIG. 8 is a flow diagram that depicts an approach for generatingintervention data.

FIG. 9 is a flow diagram that depicts an approach for generating areport.

FIG. 10 is a block diagram that depicts an arrangement for performingtranslation/transcription using multiple translation/transcriptionservices.

FIG. 11 is a block diagram that depicts an arrangement for performingtranslation/transcription using multiple translation/transcriptionservices.

FIG. 12 depicts example data that may be included in selection data.

FIG. 13 is a flow diagram that depicts an approach for generatingtranslation/transcription data using multiple translation/transcriptionservices, according to an embodiment.

FIG. 14 is a block diagram that depicts examples oftranslation/transcription data (TTD) and resultingtranslation/transcription data.

FIG. 15 is a block diagram that depicts an example arrangement on whichan embodiment may be implemented.

FIG. 16 is a flow diagram that depicts an approach for providingmulti-language support for meetings in which the participants have atleast two different native languages.

FIG. 17A is a block diagram that depicts an arrangement for managingelectronic meetings.

FIG. 17B is a block diagram that depicts an example implementation of aninteractive whiteboard appliance.

FIG. 17C is a block diagram that depicts example contents of meetinginformation in the form of a table, where each row corresponds to aparticular electronic meeting.

FIG. 17D is a block diagram that depicts a meeting arrangement thatincludes an interactive whiteboard appliance and sensors located withina room.

FIG. 18 is a flow diagram that depicts an approach for using persondetection and person identification in the context of an electronicmeeting with an interactive whiteboard appliance.

FIG. 19 is a block diagram that depicts an example implementation ofattendance tracking information for an individual named “Tom J.”

FIG. 20 is a flow diagram that depicts an approach for performingmeeting attendance tracking for an interactive whiteboard applianceaccording to an embodiment.

FIG. 21 depicts an example computer system upon which embodiments may beimplemented.

FIG. 22 is a block diagram that depicts example interactive whiteboardappliance usage information that includes interactive whiteboardappliance settings, meeting information, meeting services, and controlsystem settings.

FIG. 23 is a block diagram that depicts an example computer system uponwhich embodiments may be implemented.

While each of the drawing figures depicts a particular embodiment forpurposes of depicting a clear example, other embodiments may omit, addto, reorder, and/or modify any of the elements shown in the drawingfigures. For purposes of depicting clear examples, one or more figuresmay be described with reference to one or more other figures, but usingthe particular arrangement depicted in the one or more other figures isnot required in other embodiments.

DETAILED DESCRIPTION

In the following description, for the purposes of explanation, numerousspecific details are set forth in order to provide a thoroughunderstanding of the present disclosure. It will be apparent, however,that the present disclosure may be practiced without these specificdetails. In other instances, well-known structures and devices are shownin block diagram form in order to avoid unnecessarily obscuring thepresent disclosure. Modifiers such as “first” and “second” may be usedto differentiate elements, but the modifiers do not necessarily indicateany particular order.

-   I. Overview-   II. Architecture    -   A. Meeting Intelligence Apparatus    -   B. Network Infrastructure    -   C. Participant Nodes-   III. Meeting Rules Templates-   IV. Electronic Meetings    -   A. Meeting Creation    -   B. Meeting Agenda Creation    -   C. Meeting Participant Selection        -   i. Suggested Meeting Participants        -   ii. Subject Matter Experts (SMEs)        -   iii. Meeting Rules Templates        -   iv. Relevance Scores        -   v. Additional information-   V Real-Time Processing    -   A. Meeting Management        -   i. Agenda Management        -   ii. Points of Agreement, Action Items and New Meeting            Requests        -   iii. Sentiment Analysis    -   B. Information Retrieval    -   C. Meeting Content Supplementation    -   D. Meeting Content Metadata Generation-   VI. Post-Meeting Processing    -   A. Meeting Content Analysis    -   B. Meeting Results    -   C. Participant Analysis    -   D. Action Item Management-   VII. Process Overview    -   A. Generating Intervention Data    -   B. Generating Reports-   VIII. Translation and Transcription    -   A. Architecture    -   B. Selection Data    -   C. Generating Translation/Transcription Data    -   D. Supplemental Processing    -   E. Multi-Language Electronic Meetings    -   F. Meeting Summaries-   IX. Person Detection, Person Identification and Meeting Start    -   A. Architecture    -   B. Meeting Information    -   C. Identity Information    -   D. Meeting Arrangement    -   E. Person Detection, Person Identification and Meeting Start-   X. Attendance Tracking, Presentation of Content, Meeting Services,    and Agenda Extraction    -   A. Attendance Tracking    -   B. Presentation of Content    -   C. Meeting Services    -   D. Agenda Extraction-   XI. Learning-   XII. Implementation Examples

I. Overview

Artificial intelligence is introduced into an electronic meeting contextto perform various tasks before, during, and/or after electronicmeetings. The tasks may include a wide variety of tasks, such as agendacreation, participant selection, real-time meeting management, meetingcontent supplementation, and post-meeting processing. The artificialintelligence may analyze a wide variety of data such as data pertainingto other electronic meetings, data pertaining to organizations andusers, and other general information pertaining to any topic. Capabilityis also provided to create, manage, and enforce meeting rules templatesthat specify requirements and constraints for various aspects ofelectronic meetings. Embodiments include improved approaches fortranslation and transcription using multiple translation/transcriptionservices. Embodiments also include using sensors in conjunction withinteractive whiteboard appliances to perform person detection, personidentification, attendance tracking, and improved meeting start.Embodiments further include improvements to the presentation of contenton interactive whiteboard appliances, providing meeting services formeeting attendees, agenda extraction, and learning to aid in creatingnew electronic meetings. The approaches described herein improve theefficiency of creating and conducting electronic meetings, enforcingelectronic meeting rules and policies, allowing decisions to be mademore quickly, and providing higher confidence in the usefulness ofelectronic meetings.

II. Architecture

FIG. 1A-C depict example computer architectures upon which embodimentsmay be implemented. FIG. 1A-C include various examples of an arrangement100 that supports electronic meetings. Arrangement 100 includes ameeting intelligence apparatus 102 and one or more nodes 104A-N,communicatively coupled via network infrastructure 106. Nodes 104A-N areassociated with a plurality of electronic meeting participants 108A-N,also referred to herein as “participants.” Some elements depicted inFIG. 1A are not depicted in other figures for purposes of explanation,but these elements may be included in the elements of the other figures.

Electronic meetings supported by arrangement 100 may be any type ofelectronic meetings conducted in any form using any computing platform.Examples of electronic meetings include, without limitation,audioconferencing sessions, videoconferencing sessions, desktop sharingsessions, any combination of the foregoing, and/or any other electronicmeetings involving the sharing of data between network infrastructure106 and at least one node 104A. Participants 108A-N may be located inphysically-disparate locations or co-located in the same physicallocation. For example, FIG. 1A-B depict arrangement 100 in whichparticipants 108A-N are located in different physical locations, yetparticipant in an electronic meeting via network infrastructure 106.FIG. 1C depicts arrangement 100 in which participants 108A-N are locatedin physical proximity to each other such that they may communicate witheach other without network infrastructure 106. However, networkinfrastructure 106 may enable participants 108A-N to interact withmeeting intelligence apparatus 102, which receives input data fromand/or sends output data to node 104A. Embodiments are also applicableto electronic meetings in which some participants 108A-N are inphysically-disparate locations and some are co-located at the samephysical location.

In an embodiment, arrangement 100 involves a network of computers. A“computer” may be one or more physical computers, virtual computers,and/or computing devices. A computer may be a client and/or a server.Any reference to “a computer” herein may mean one or more computers,unless expressly stated otherwise. Each of the logical and/or functionalunits depicted in any of the figures or described herein may beimplemented using any of the techniques further described herein inconnection with FIG. 21.

A. Meeting Intelligence Apparatus

In an embodiment, meeting intelligence apparatus 102 is implemented byone or more computing devices configured with artificial intelligence.The one or more computing devices may be special-purpose computingdevices dedicated to providing artificial intelligence to electronicmeetings, or generic computing devices executing one or more artificialintelligence services, such as artificial intelligence service 110 (FIG.1A), which provide artificial intelligence to electronic meetings. Thus,meeting intelligence apparatus 102 may be implemented using anycombination of computer hardware, computer software, and/or computerfirmware. Examples of artificial intelligence service 110 include,without limitation, IBM Watson and other question answering (QA)systems/services. Meeting intelligence apparatus 102 may always beavailable (e.g., involve continuously running processes) or may beavailable on demand (e.g., be powered on when needed). Meetingintelligence apparatus 102 may be replicated over multiple computingdevices such that at any point in time, at least one computing devicecan provide meeting intelligence services.

Meeting intelligence apparatus 102 may access meeting content data as ifit were a node associated with a participant in an electronic meeting.Thus, meeting intelligence apparatus 102 may access any meeting contentdata that is transmitted from any of the one or more nodes 104A-Ninvolved in an electronic meeting. For example, meeting intelligenceapparatus 102 may monitor, collect, and/or analyze all datatransmissions during an electronic meeting. Meeting intelligenceapparatus 102 may analyze meeting content data using any of a number oftools, such as speech or text recognition, voice or face identification,sentiment analysis, object detection, gestural analysis, thermalimaging, etc. Based on analyzing the meeting content data and/or inresponse to requests, for example, from electronic meeting applications,meeting intelligence apparatus 102, either alone or in combination withone or more electronic meeting applications, performs any of a number ofautomated tasks, including performing one or more actions with respectto an electronic meeting, such as creating an electronic meeting,providing a translation, responding to an information request,determining suggested agenda items and suggested meeting participants,performing various types of real-time processing with respect to anelectronic meeting, such as managing the flow of an electronic meeting,processing agenda items, creating actions items, etc., and variouspost-meeting processing, such as generating a report, etc.

Meeting intelligence apparatus 102 may be located at a number ofdifferent locations relative to network infrastructure 106. Referring toFIGS. 1A and 1C, meeting intelligence apparatus 102 is located outsidenetwork infrastructure 106. Referring to FIG. 1B, meeting intelligenceapparatus 102 is collocated with at least some of network infrastructure106. Artificial intelligence service 110 may include one or moreapplication program interfaces (APIs) that may be used by electronicmeeting applications to access functionality provided by artificialintelligence service 110.

In an embodiment, meeting intelligence apparatus 102 is communicativelycoupled to a meeting data repository (not depicted). The meeting datarepository may be part of meeting intelligence apparatus 102, or may belocated separate from meeting intelligence apparatus 102. The meetingdata repository may be a database, data structure, a configuration file,and/or any other system that stores data related to one or moreelectronic meetings. Like meeting intelligence apparatus 102, themeeting data repository may be located at different locations relativeto network infrastructure 106, for example, on one or more computingdevices internal or external to network infrastructure 106. The meetingdata repository stores data pertaining to any number of electronicmeetings, and may include data for prior electronic meetings, currentelectronic meetings, and future electronic meetings. Examples of datafor prior, current and future electronic meetings include, withoutlimitation, meeting agendas, meeting participant information, meetinginvitation information, meeting transcripts, minutes and notes, actionitems, etc., as well as results data that includes the results ofanalyzing any of the foregoing types of data. Meeting data may becollected and stored by meeting intelligence apparatus 102, nodes104A-N, or both. This allows meeting intelligence apparatus 102 toprovide the services of a librarian for meeting-related data, as well asthe other functionality described herein. Electronic meeting data mayalso be generated, stored and managed by other entities, such as otherprocesses.

In an embodiment, meeting intelligence apparatus 102 is communicativelycoupled to any of a number of external data sources (not shown), such aswebsites, other data available via the World Wide Web, databases managedby Salesforce, Oracle, SAP, Workday, or any entity other than the entitymanaging meeting intelligence apparatus 102. Meeting intelligenceapparatus 102 may be communicatively coupled to the external datasources via network infrastructure 106. The external data sources mayprovide meeting intelligence apparatus 102 with access to any of avariety of data, meeting-related or otherwise.

B. Network Infrastructure

Network infrastructure 106 may include any number and type of wired orwireless networks, such as local area networks (LANs), wide areanetworks (WANs), the Internet, etc. Network infrastructure 106 may alsoinclude one or more computing devices, such as one or more servercomputers, load-balancing computers, cloud-based computers, datacenters, storage devices, and/or any other special-purpose computingdevices. For example, network infrastructure 106 may include a UnifiedCommunication System (UCS) Service Platform by Ricoh Company Ltd.,and/or any other computer(s) that manage(s) electronic meetings.

C. Participant Nodes

Each node of the one or more nodes 104A-N is associated with one or moreparticipants 108A-N. Each participant is a person who participates in anelectronic meeting. Each node processes data transmission betweennetwork infrastructure 106 and at least one participant. Multiple nodes104A-N may be communicatively coupled with each other using any of anumber of different configurations. For example, multiple nodes may becommunicatively coupled with each other via a centralized server or viaa peer-to-peer network.

In an embodiment, a node includes a computing device that executes anelectronic meeting application 112. As used herein, the term“application” refers to any type of process. The node may include aspecial-purpose computer, such as Ricoh UCS P3500, or a general-purposecomputer that executes a special-purpose application, such as Ricoh UCSApp. The node may also include any of a number of input/outputmechanisms, such as a camera, a microphone, and an electronicwhiteboard. For example, the node may include a smartphone with GPScapability, a camera, a microphone, an accelerometer, a touchscreen,etc. Example implementations of a node include, without limitation, adesktop computer, a laptop computer, a workstation, a tablet computingdevice, a personal digital assistant, an interactive whiteboard (IWB)appliance, a mobile phone, or other mobile computing device. Theinput/output mechanisms may include a participant interface, such as agraphical user interface (GUI). Electronic meeting applications maycreate and manage data that supports electronic meetings, and the datamay be stored locally on nodes 104A-N, remotely, or both locally andremotely, depending upon a particular implementation. Electronic meetingapplication 112 is implemented by one or more processes that allow acorresponding node to participate in electronic meetings. Electronicmeeting application 112 is configured to interact with one or more otherelectronic meeting applications on other computing devices and/or one ormore electronic meeting managers or servers to participate in electronicmeetings. For example, electronic meeting application 112 may interactwith an electronic meeting manager to perform various functionspertaining to electronic meetings. This may include, for example,scheduling meetings, creating agendas and specifying content to beshared/discussed, inviting participants, etc. Electronic meetingapplication 112 is also configured to access functionality provided byartificial intelligence service 110, via one or more APIs provided byartificial intelligence service 110, to incorporate artificialintelligence into various aspects of electronic meetings, as describedin more detail hereinafter.

III. Meeting Rules Templates

According to one embodiment, functionality is provided to create andmanage meeting rules templates for electronic meetings. A meeting rulestemplate specifies a set of rules to be applied to any aspect of anelectronic meeting, i.e., before, during, and after an electronicmeeting. As used herein, the term “rule” refers to a requirement and/ora constraint. For example, a meeting rules template may define itemsthat are required to be included on an agenda, participants that arerequired in an electronic meeting, that a decision maker must bedesignated, one or more aspects of how an electronic meeting isconducted, e.g., via time limits, etc., procedural rules that govern anelectronic meeting, and items to be included in meeting results. Meetingrules templates may also specify data inputs for an electronic meeting,such as data streams, electronic documents, etc. Meeting rules templatefunctionality may be implemented, for example, by an electronic meetingapplication executing on a node 104A-N.

Meeting rules may be specified by an organization, e.g., via bylaws, orby entities external to organizations, such as governmental, judicial orlaw enforcement entities. One non-limiting example is organizationalrules or bylaws that require at least a specified number of participantsattend an electronic meeting to satisfy a quorum requirement for votingpurposes. The participants may include regular members, board members,voting members, etc. Another example is a time constraint (minimum ormaximum) for discussion of a particular agenda item. Yet another exampleis a requirement that documentation for an electronic meeting, such asmeeting minutes, include specified information, such as start and endtimes, the names of board members in attendance, decisions that weremade on action items, etc. Requirements and constraints may also specifythat a particular meeting procedure be used to conduct an electronicmeeting. One non-limiting example of a meeting procedure is RobertsRules of Order.

As described in more detail hereinafter, according to one embodiment, anelectronic meeting application allows a user to designate one or moremeeting rules templates for an electronic meeting. The electronicmeeting application, in conjunction with meeting intelligence apparatus102, enforces the rules specified by the designated meeting rulestemplates before, during, and after electronic meetings. This mayinclude, for example, ensuring that an agenda for an electronic meetingcontains required items, that required participants are included in anelectronic meeting, that required data inputs are provided, that theelectronic meeting is conducted in accordance with the requirements andconstraints, and that meeting results satisfy the requirements andconstraints specified by a meeting rules template.

FIG. 2A depicts an example electronic meeting screen 200 displayed by anelectronic meeting application at a node. Electronic meeting screen 200includes controls 202 for accessing meeting rules templates, electronicmeetings, meeting reports and action items. Electronic meeting screen200 may represent an initial meeting screen that is displayed after anelectronic meeting application is invoked on a node 104A-N.

FIG. 2B depicts an example meeting rules templates screen 204 displayedin response to a user selecting the “Meeting Rules Templates” optionfrom controls 202 in FIG. 2A. Meeting rules templates screen 204 allowsa user to view, edit and delete existing meeting rules templates, andcreate new meeting rules templates. Meeting rules templates screen 204depicts existing meeting rules templates displayed by department, withina business organization, and by topic. In the example depicted in FIG.2B, the departments include Legal, Engineering, Software QualityAssurance (SQA), Finance and Management, with one or more identifiedmeeting rules templates for each department. The particular departmentsincluded in FIG. 2B are provided for explanation purposes, and theorganization of meeting rules templates may vary depending upon aparticular implementation. For example, meeting rules templates may beorganized and displayed by any type of logical entity.

In the example depicted in FIG. 2B, a user may select an existingmeeting rules template and then edit or delete the selected meetingrules template by selecting an “Edit” control or a “Delete” control,respectively, from controls 206. According to one embodiment,permissions may be used to control which users have the authority toedit or delete a meeting rules template. For example, permissions datamay be maintained for meeting rules templates, where the permissionsdata specifies users who are authorized to edit and delete particularmeeting rules templates. This may be useful for meeting rules templates,such as the “Board of Directors” meeting rules template that may containsensitive information. A user may also create a new meeting rulestemplate by selecting a “New” control from controls 206. According toone embodiment, an electronic meeting application may include a wizardthat provides step-by-step assistance for users to create new meetingrules templates. This may include, for example, providing commonly-usedrules with controls that allow a user to select and de-select individualrules, as well as add custom rules. A “Back” control allows a user toreturn to the electronic meeting screen 200 of FIG. 2B. A search tool208 is also provided to allow a user to search for existing rulestemplates, for example, by keyword.

FIG. 2C depicts meeting rules templates screen 204 that displays the“Board of Directors” meeting rules template for the “Management”department selected by a user from the meeting rules templates screen204 in FIG. 2B. The Board of Directors meeting rules template specifiesrules that must be satisfied before, during, and after Board ofDirectors meetings. These rules specify that a Board of Directorsmeeting may have a maximum number of 10 meeting participants, a minimumnumber of four board members are required, a designated decision makeris required, certain agenda items are required, including a financialreview, short term strategic planning, and long term strategic planning,and a maximum amount of time of 15 minutes may be spent on each agendaitem. The rules also specify that Board of Directors meetings must beconducted in accordance with Roberts Rules of Order and meeting minutesmust be generated that include a start and end time of the meeting, alist of the Board members in attendance, and the results of votes on allmotions. The rules further specify that Board of Directors meetings mustinclude as data inputs, Q4 Financials and a Sales Projection Report. Theparticular data inputs may vary, depending upon the requirements of aparticular meeting.

The meeting rules depicted in FIG. 2C are provided for explanationpurposes, and meeting rules may vary, depending upon a particularimplementation. Controls 210 allow a user to add, edit or delete rules,save rules, or return to the meeting rules templates screen 204 of FIG.2B by selecting the “Back” option.

Meeting rules may be manually created by users, for example, by anadministrator or end user. According to one embodiment, meeting rulesmay be created with the assistance of meeting intelligence apparatus102. For example, when creating a new rule, via the “Add” control fromcontrols 210, the meeting rules templates screen 204 may provide anoption that allows a user to enter query terms for a particular type ofmeeting for which the user is interested in defining meeting rules.Meeting intelligence apparatus 102 processes the query terms andpresents results that include example meeting rules that may beappropriate for the particular type of meeting. The example meetingrules may be obtained, for example, based upon industry-specificpractices, or “best practices.” For example, the widely-accepted “bestpractices” for a particular type of engineering meeting may include asmall number of participants, e.g., less than five, and that input fromall meeting participants is required.

IV. Electronic Meetings

According to one embodiment, an electronic meeting application executingon a node 104A-N includes functionality that allows user to manageelectronic meetings. FIG. 2D depicts an electronic meeting screen 212displayed by an electronic meeting application in response to a userselecting the “Electronic Meetings” option from controls 202 in FIG. 2A.Electronic meeting screen 212 may be implemented, for example, by one ormore Web pages which, when processed on a computing device, causeelectronic meeting screen 212 to be displayed on the computing device.

Electronic meeting screen 212 includes a content window 213 thatincludes content 214 for a current electronic meeting, which mayrepresent a videoconferencing session, a desktop sharing session, amessaging session, an audio conferencing session, etc. Electronicmeeting screen 212 also includes a participant window 216 thatidentifies participants of the current electronic meeting, and an agendawindow 218 that displays an agenda for the current electronic meeting.Agenda window 218 includes a visual indicator 220 that identifies acurrent agenda item and a timer that indicates the time remaining forthe current agenda item. The management of agenda items using meetingintelligence apparatus 102 is discussed in more detail hereinafter.Electronic meeting screen 212 also includes meeting controls 222 thatprovide access to various functionality with respect to the currentelectronic meeting. As described in more detail hereinafter, thisincludes functionality pertaining to electronic meeting management,content, agendas, and participants. A message window 224 providesmessages generated with the assistance of meeting intelligenceapparatus, which is also described in more detail hereinafter.Electronic meeting screen 212 may include additional elements or fewerelements depending upon a particular implementation, and embodiments arenot limited to the example depicted in FIG. 2D.

FIG. 2E depicts an electronic meeting management screen 230 displayed byan electronic meeting application in response to a user selecting acontrol from meeting controls 222. Electronic meeting management screen230 includes an existing meeting portion 232 that displays informationand provides controls to view and join existing electronic meetings, anda new meeting portion 234 that includes controls that allow a user tocreate a new electronic meeting. According to one embodiment,permissions may be used to control the ability for users to view and/orjoin an electronic meeting. Permissions may be implemented in a widevariety of ways that may vary depending upon a particularimplementation. For example, in response to a user selecting the “Join”option for a particular electronic meeting via existing meeting portion232, a request to join the particular electronic meeting may begenerated and transmitted to the organizer of the electronic meeting. Inresponse to the meeting organizer approving the request, the user is maybe notified that their request has been granted and the user is allowedto join the particular electronic meeting. As an alternative, thisprocess may be automated based upon a user's permissions. For example,the permissions of the user requesting to join the particular meetingare compared to permissions specified for the particular electronicmeeting and the user is allowed to join the particular electronicmeeting if the user has the necessary permissions. The use ofpermissions in the foregoing examples is helpful for controlling accessto electronic meetings that may include sensitive or confidentialinformation. Electronic meeting management screen 230 includes a “Back”control to return to electronic meeting screen 200.

In the example depicted in FIG. 2E, existing meetings portion 232depicts existing electronic meetings for logical entities within abusiness organization that include a Pluto Project, a Legal Department,a Finance Department, and Management, with corresponding controls toview or join any of these electronic meetings. Embodiments are notlimited to these example logical entities, and any type of logicalentities may be used. A user may select the view control to viewinformation for a particular existing electronic meeting.

FIG. 2F depicts a view electronic meeting screen 240 displayed inresponse to a user selecting the “View” option for the Code ReviewMeeting for the Pluto Project displayed in existing meetings portion232. In the example depicted in FIG. 2F, view electronic meeting screen240 displays information 242 for the Code Review Meeting for the PlutoProject, including date, time duration, location and attendees. Theinformation 242 also includes electronic meeting rules 244 and an agenda246. As described in more detail hereinafter, electronic meeting rules244 and agenda 246 may be generated with the assistance of artificialintelligence provided by meeting intelligence apparatus 102. In theexample depicted in FIG. 2F, electronic meeting rules 244 are defined bya particular meeting rules template, namely, the Engineering—Code Reviewmeeting rules template depicted in FIG. 2B. This meeting rules templatespecifies a maximum number of 10 participants, that the meeting mustinclude a designated decision maker, a maximum of 20 minutes may bespent on each topic, and no meeting minutes are required.

Agenda 246 specifies topics to be addressed during the electronicmeeting and includes a revised dynamic linked library (DLL) for thesatellite link, driver code for the direct memory access (DMA) board, amilestone status review, software testing with hardware (h/w) in thevacuum chamber, the software testing schedule, and software integrationplanning. A user who has the necessary permissions, such as the owner orhost of the Code Review Meeting for the Pluto Project, may edit theinformation 242 for the electronic meeting. For example, a user mayselect the “Duration” attribute and change the value from 1 hour to 45minutes. Information 242 also includes a control that allows a user tojoin the electronic meeting. Selecting the “Join” option, either fromview electronic meeting screen 240, or the electronic meeting managementscreen 230 (FIG. 2E), causes a request for the user to join the meetingto be sent to the meeting owner for processing.

A. Meeting Creation

Electronic meetings may be created manually by users, or may be createdwith the assistance of artificial intelligence provided by meetingintelligence apparatus 102. Returning to FIG. 2E, new meeting portion234 includes controls that allow a user to specify, for a new electronicmeeting, a meeting name, a meeting owner, a meeting subject, and ameeting type. The controls also allow a user to select one or moremeeting rules templates, a date and time, and a location for the newelectronic meeting. The location may correspond to the physical locationof a computing device of the electronic meeting owner or host. A singlemeeting rules template is depicted in FIG. 2E for purposes ofexplanation, but electronic meetings may have any number ofcorresponding meeting rules templates. Embodiments are not limited tothe information depicted in FIG. 2E, and new meeting portion 234 mayallow a user to specify other information for new electronic meetings,depending upon a particular implementation.

According to one embodiment, missing information for new meetings may beprovided by meeting intelligence apparatus 102 based upon an analysis ofvarious types of data. In the example depicted in FIG. 2E, a meetingowner was not specified. Meeting intelligence apparatus 102 maydetermine, based upon an analysis of prior meetings for the Plutoproject, such as a first code review meeting, that Bob. H is a goodcandidate to be the meeting owner of the second code review meeting, andthe meeting owner field may be automatically populated with Bob H. Otherinformation may be considered in making the determination, such as othertypes of meetings for the same project, code review meetings for otherprojects, assigned responsibilities of project members, etc.

Missing information for a new meeting may be provided or indicated tothe electronic meeting application executing on node 104A-N andautomatically included in new meeting information. Missing informationmay be presented in a manner to visually indicate that the informationwas automatically provided, for example, via highlighting, coloring,special effects, etc., and a user may be given an opportunity to accept,reject, or edit the missing information that was automatically provided.Meeting intelligence apparatus 102 may also provide multiple candidatesfor missing information.

In the prior example, meeting intelligence apparatus 102 may determinethat both Bob H. and another person Susan G. are good candidates to bethe meeting owner of the second code review meeting. The two candidatesmay be presented as suggestions for designation as the meeting owner,and a user given an opportunity to select one, both, or none of thechoices, unless the electronic meeting application does not allowmultiple meeting owners, in which case only one of Bob H. and Susan G.may be selected as the meeting owner. Suggestions may be presented inranked order, based upon a confidence score, and the confidence scorefor each choice may be displayed to provide additional information tothe user making the selection.

FIG. 2G depicts electronic meeting management screen 230 with a pop-upwindow 239 that provides suggestions for missing meeting information,which in the present example is the missing meeting owner. In thisexample it is presumed that the creator of the new electronic meetingdid not specify a meeting owner. The suggestions for the meeting ownerare determined by the electronic meeting application in conjunction withmeeting intelligence apparatus 102. In this example, the suggestedmeeting owners Bob H. and Susan G. are displayed in pop-up window 239,and controls are provided to accept or reject one or both of thesuggested meeting owners, along with controls for saving the informationfor the new electronic meeting, or canceling and not saving theinformation. A confidence score generated by meeting intelligenceapparatus 102 is optionally displayed for each suggestion. Theconfidence score represents a likelihood that the correspondingsuggestion will be selected by the user.

Additional information is optionally provided with the suggestions toaid the user in making a selection of one, both or none of the suggestedmeeting owners. The additional information may be information thatmeeting intelligence apparatus 102 considered in making the suggestions,or may be other information that is deemed likely to be helpful to usersin considering the suggested missing information, i.e., the missingmeeting owner in the present example. As depicted in FIG. 2G, theadditional information for Bob H. indicates that Bob H. was the owner ofprior code review meetings for the Pluto project, making him a candidatefor the meeting owner. Susan G. is a member of the Pluto project teamand was the owner of code review meetings for other projects, whichcombined makes Susan G. a candidate for the meeting owner of the secondcode review meeting for the Pluto project. This example illustrates howcognitive computing may be useful in providing suggestions for missinginformation for a new electronic meeting. This approach may be appliedto any missing information, depending upon a particular implementation.

Electronic meetings may also be automatically created during electronicmeetings by an electronic meeting application executing on a node, inconjunction with meeting intelligence apparatus 102. According to oneembodiment, meeting intelligence apparatus 102 detects, during anelectronic meeting, one or more cues that indicate that a new meetingshould be created. For example, meeting intelligence apparatus 102 maydetect, in meeting content data, an explicit command, such as a naturallanguage request, from a meeting participant to create a new meeting.Meeting intelligence apparatus 102 may be configured to recognizeparticular commands and formats, e.g., “SystemOne: create new meeting.”

Meeting intelligence apparatus 102 may also be configured to create newmeetings based upon implicit commands. Implicit commands or requests arestatements made during an electronic meeting that indicate a desire fora new meeting, even though the statements may not include an explicitspecified command or request for a new meeting. For example, meetingintelligence apparatus 102 may detect, in meeting content data,statements made as part of a conversation to indicate that anothermeeting may be useful in discussing a particular topic. Examples of suchstatements include “It would be good to discuss this in a separatemeeting” and “Let's setup another meeting to discuss this further.”These statements can be characterized as representing an implied requestfor a new electronic meeting. Cognitive computing functionality providedby meeting intelligence apparatus 102 may be used to analyze statementsmade during an electronic meeting to determine whether an implicitcommand or request for a new electronic meeting has been made during anelectronic meeting. Meeting intelligence apparatus 102 may then informthe electronic meeting application that a new electronic meeting shouldbe created, as described in more detail hereinafter.

Meeting intelligence apparatus 102 may be further configured to create anew electronic meeting based upon sentiment analysis of a currentelectronic meeting that indicates that a new meeting may be appropriateor helpful in a particular situation. For example, sentiment analysis ofa current electronic meeting may indicate that significant disagreementand/or hostility exists between participants in a meeting. Sentimentanalysis may use various cues that occur in speech during an electronicmeeting, such as tone of voice, volume of voice, velocity of speech,lack of pauses in speech, profanity, sounds such as grunts, exhalationof air, etc. The use of sentiment analysis is described in more detailhereinafter.

In response to determining that a new meeting is appropriate, based uponan explicit command, an implicit command, or sentiment analysis, meetingintelligence apparatus 102 may directly instruct an electronic meetingapplication to request a new meeting, or include an instruction orrequest in meeting content data or meeting metadata that is provided tothe electronic meeting application. In response to receiving aninstruction or request, the electronic meeting application may provide avisual and/or audible notification to indicate a new meeting request.The electronic meeting application may display information for the newelectronic meeting determined from information in the current electronicmeeting. For example, the electronic meeting application, in conjunctionwith meeting intelligence apparatus 102, may determine information forthe new electronic meeting based upon information in the currentelectronic meeting. For example, a statement may be made during thecurrent electronic meeting, “We should have a separate meeting for thisnext Tuesday,” and a proposed date and time would be included with theinformation for the new electronic meeting. The electronic meetingapplication may, in conjunction with meeting intelligence apparatus 102,query a user for additional information about the new electronicmeeting, for example, information that is deemed to be missing for thenew electronic meeting.

B. Meeting Agenda Creation

Electronic meeting agendas may be created manually by users and may becreated with the assistance of artificial intelligence provided bymeeting intelligence apparatus 102. According to one embodiment, meetingintelligence apparatus 102 participates in the creation of electronicmeeting agendas by providing suggested items to be included on anelectronic meeting agenda. The electronic meeting application mayrequest that meeting intelligence apparatus 102 provide suggested agendaitems for an electronic meeting. The request for suggested agenda itemsmay include data that specifies a plurality of attributes of the newelectronic meeting. For example, a request for suggested agenda itemsmay include any of the data specified by a user in new meeting portion234 of electronic meeting management screen 230. In response to thisrequest, meeting intelligence apparatus 102 analyzes various types ofinformation and provides suggested agenda items to the electronicmeeting application, which displays the suggested items for a user, andallows a user to select suggested items to be included in an electronicmeeting agenda.

Returning to FIG. 2E, new meeting portion 234 includes a Meeting Agendacontrol 236 that allows a user to create a meeting agenda for a newelectronic meeting. FIG. 2H depicts an agenda creation screen 250generated in response to a user selecting Meeting Agenda control 236from electronic meeting management screen 230. For example, in responseto a user selecting Meeting Agenda control 236 from electronic meetingmanagement screen 230, the electronic meeting application may causeagenda creation screen 250 to be generated and displayed.

Agenda creation screen 250 includes agenda information 252 for the newelectronic meeting, which in this example is the second code reviewmeeting for the Pluto project. Agenda information 252 includesuser-specified agenda items 254 that may be manually specified by auser. User-specified agenda items 254 may be specified, for example, asa list of agenda items that a user wishes to be included on the meetingagenda for the new electronic meeting. In the example depicted in FIG.2H, user-specified agenda items 254 include three agenda items:Additional Software Requirements from Vendor, Graphical User InterfaceUpdate, and Software Integration Planning. These agenda items may bespecified by a user typing the agenda items into user-specified agendaitems 254, or by copying and pasting information from otherapplications. User-specified agenda items 254 may be added, edited, anddeleted, and the changes saved via selection of the Save control 258.

Agenda information 252 also includes suggested agenda items 256 that aregenerated with the assistance of meeting intelligence apparatus 102.Suggested agenda items 256 may be automatically generated and displayedin response to a user selecting Meeting Agenda control 236 when agendacreation screen 250 is displayed. Alternatively, agenda creation screen250 may include a control for requesting the generation and display ofsuggested agenda items 256. A user may switch between user-specifiedagenda items 254 and suggested agenda items 256 by selecting eachcorresponding area on agenda creation screen 250, for example, by usinga selection device such as a mouse.

Suggested agenda items 256 are topics for discussion that are determinedto be relevant and appropriate for a particular new electronic meeting.The topics may be topics that have been scheduled for discussion, oractually discussed, in other electronic meetings, or they may be newtopics. For example, suggested agenda items 259 may include topics thatappear in multiple electronic documents related to the subject matter ofa new electronic meeting. To generate suggested agenda items 256,meeting intelligence apparatus 102 may analyze agendas of other prior orcurrent electronic meetings, minutes or other written records of otherelectronic meetings, other documentation, or any other information. Thismay include information for the same entity associated with theelectronic meeting, other entities, or information for a particularindustry or context. Examples of information analyzed by meetingintelligence apparatus 102 includes, without limitation, meeting agendasor other documents for other electronic meetings of the same meetingtype for the same project, of the same meeting type for other projectswithin an entity, such as a business or other organization, of adifferent, but related meeting type, for the same project, or for otherprojects, etc. this may include, for example, search for electronicdocuments that mention the current project, which in the present exampleis the Pluto Project. Other examples include, without limitation,meeting agendas for other organizations within the same industry orcontext. Information beyond meeting agendas may also be evaluated.Examples of other information include, without limitation,specifications, memoranda, and other documents.

Suggested agenda items 256 may be organized and presented to a user inany manner that may vary depending upon a particular implementation. Fora large number of suggested agenda items 256, visually organizing thesuggested agenda items on a user interface may provide a more favorableuser experience than merely listing all available suggested agenda items256. FIG. 2H depicts categories of suggested agenda items 256:Uncompleted Agenda Items From Prior Meeting(s), Action Items From PriorMeeting(s), Agenda Items Based Upon Meeting Rules Template, and Other.Each of the categories may be selected to view the particular agendaitems within each category.

FIG. 2I depicts suggested agenda items for each category of suggestedagenda items depicted in FIG. 2H. Organizing suggested agenda items bycategory may be more useful to some users than listing suggesting agendaitems in random order, although embodiments are not limited toorganizing suggested agenda items 256 by category, and other approachesmay be used such as alphabetical order, etc.

Agenda items in the Uncompleted Agenda Items and Action Items From PriorMeeting(s) categories may be determined by meeting intelligenceapparatus 102 first identifying one or more prior electronic meetingsthat are related to the current electronic meeting, and then analyzingelectronic documents associated with the identified one or more priorelectronic meetings. Meeting intelligence apparatus 102 may identify theone or more prior electronic meetings, which in this example is theFirst Code Review Meeting, based upon a reference to the First CodeReview Meeting in documentation associated with the current electronicmeeting, i.e., the Second Code Review Meeting, or any other attributesof the current meeting. For example, the agenda for the currentelectronic meeting may explicitly reference the prior electronicmeeting. As another example, the one or more prior electronic meetingsmay be referenced by a schedule or plan. For example, both the one ormore prior electronic meetings and the current electronic meeting may beincluded on a project schedule.

Once the one or more prior electronic meetings have been identified,electronic documents for the one or more prior electronic meetings, suchas the agenda and/or meeting minutes of the prior meeting, are analyzedto determine the agenda items from the one or more prior electronicmeetings that were not completed, as well as action items that werecreated during the one or more prior electronic meetings. For example,meeting intelligence apparatus 102 may examine meeting minutes from theFirst Code Review Meeting to identify agenda items from that were notcompleted, as well as action items created during that meeting. Thoseidentified agenda items are added to the Uncompleted Agenda Items FromPrior Meeting(s) and Action Items From Prior Meeting(s) categories. Inthe example depicted in FIG. 21, the “Driver code for DMA board” and“Software Testing Schedule” agenda items were uncompleted from the FirstCode Review Meeting, and the “Verify requirements for Satlink—John E.”and “Investigate problems with DMA board—Susan R.” action items wereassigned during or after the First Code Review Meeting.

Agenda items in the Agenda Items Based Upon Meeting Rules Templatecategory may be determined by meeting intelligence apparatus 102analyzing one or more meeting rules templates specified for the newelectronic meeting. In the present example, meeting intelligenceapparatus 102 determines that the “Milestones Status Review” agenda itemis required by the Engineering—Code Review meeting rules template.Meeting intelligence apparatus 102 therefore uses the “Milestones StatusReview” agenda item as a suggested agenda item. According to oneembodiment, agenda items required by a meeting rules template are notprovided as suggested agenda items that may be selected or not selectedby a user. Instead, the agenda items that are required by a meetingrules template are automatically added to the agenda for the currentelectronic meeting. This ensures that required agenda items specified bya meeting rules template are included on the agenda, and also reducesthe number of suggested agenda items that must be reviewed by a user.

Agenda items in the Other category may be determined by meetingintelligence apparatus 102 analyzing a plurality of data items toidentify suggested agenda items that are relevant to the new electronicmeeting. The plurality of data items analyzed by meeting intelligenceapparatus 102 may include any type of data items that may vary dependingupon a particular implementation. Relevance may be determined using awide variety of algorithms and heuristics that may vary depending upon aparticular implementation. According to one embodiment, meetingintelligence apparatus 102 is configured to analyze a plurality of dataitems to identify typical agenda items for the meeting type of the newelectronic meeting. In the present example, this includes determiningtypical agenda items for code review meetings. This may includedetermining the typical agenda items for code review meetings within thesame organization, or searching beyond the current organization to otherorganizations. The search may be conducted within the same context,industry, etc., or may extend to other contexts, industries, etc.According to one embodiment, meeting intelligence apparatus 102identifies electronic documents related to one or more topics orsubjects of the new electronic meeting and then analyzes the identifiedelectronic documents to determine one or more suggested agenda items forthe Other category. In the present example, meeting intelligenceapparatus 102 determines that the “Software Testing Schedule” agendaitem is typical for code review meetings and is therefore included as asuggested agenda item. Other criteria besides meeting type may be usedto identify suggested agenda items. For example, the meeting subject maybe used as a criterion to identify suggested agenda items. In thepresent example, meeting intelligence apparatus may search the pluralityof data items to identify data items related to the Pluto Project, anddetermine suggested agenda items based upon the data items related tothe Pluto Project.

According to one embodiment, relevance scores may optionally bedetermined and displayed for suggested agenda items 256. Relevancescores may be separately determined for each of the suggested agendaitems 256 based upon various factors. As depicted in FIG. 21, relevancescores for suggested agenda items in the Uncompleted Agenda Items FromPrior Meeting(s) and Action Items From Prior Meeting(s) categories arerelatively high, which is attributable to a high correlation between thesuggested agenda items and one or more attributes of the new electronicmeeting. For example, the high correlation may be between the suggestedagenda items and one or more subjects or meeting types of the electronicmeeting. This may result in a user being more likely to select thesuggested agenda items for the new electronic meeting. Similarly,relevance scores for suggested agenda items in the Agenda Items BasedUpon Meeting Rules Templates are also relatively high since these agendaitems are required by the one or more meeting rules templates specifiedfor the new electronic meeting and are therefore determined to be highlycorrelated to one or more attributes of the new electronic meeting. Incontrast, a relevance score of 92% is determined and displayed for thesuggested agenda item “Software Testing Schedule.” This relevance scoreis lower than the relevance scores for the Uncompleted Agenda Items FromPrior Meeting(s) and Action Items From Prior Meeting(s) categories sincethe “Software Test Schedule” was not an uncompleted agenda item or anaction item from a prior meeting, but is nevertheless determined to havea high relevance score based upon, for example, 1) the meeting type ofthe new electronic meeting being a code review meeting; and 2) ananalysis of agendas for other code review meetings showed that asoftware test schedule is a common agenda item for code review meeting.

Suggested agenda items may be displayed in a generalized form toidentify the topics for discussion, while allowing a user to edit andtailor the suggested agenda items for a particular electronic meeting.Controls, in the form of checkboxes, are provided to allow a user toselect and deselect individual agenda items. Selecting a particularagenda item causes the particular agenda item to be included in theagenda for the new electronic meeting. According to one embodiment,agenda creation screen 250 provides the capability for users to editsuggested agenda items 256, for example, by selecting a particularsuggested agenda item using a pointing device such as a mouse, and thenedit the text of the particular suggested agenda item. This may beuseful, for example, to specify changes or clarifications to an agendaitem, which is commonly done on the context of electronic meetings. Oncethe user has completed editing (if applicable) and selecting suggestedagenda items 256, a selection of the “Save” control from controls 258causes the agenda information 252 for the new electronic meeting to besaved. Selecting of the “Back” control from controls 258 causes controlto be returned to the electronic meeting management screen 230.

Meeting Participant Selection

Electronic meeting participants may be selected manually by users andaccording to one embodiment, the selection of meeting participants isassisted by artificial intelligence provided by meeting intelligenceapparatus 102. Meeting intelligence apparatus 102 participates in theselection of electronic meeting participants by providing suggestedmeeting participants for an electronic meeting. In addition, asdescribed in more detail hereinafter, constraints and requirements formeeting participants specified by a meeting rules template designatedfor an electronic meeting may be enforced by electronic meetingapplications.

The electronic meeting application may request that meeting intelligenceapparatus 102 provide suggested meeting participants for an electronicmeeting. The request for suggested meeting participants may include datathat specifies a plurality of attributes of the electronic meeting. Forexample, a request for suggested agenda items may include any of thedata specified by a user in new meeting portion 234 of electronicmeeting management screen 230. In response to this request, meetingintelligence apparatus 102 analyzes various types of information andprovides suggested meeting participants to the electronic meetingapplication, which displays the suggested meeting participants for auser, and allows a user to select suggested meeting participants to beincluded in an electronic meeting.

Returning to FIG. 2E, new meeting portion 234 includes a MeetingParticipants control 238 that allows a user to select participants for anew electronic meeting. FIG. 2J depicts a participant selection screen260 generated in response to a user selecting Meeting Participantscontrol 238 from electronic meeting management screen 230. Participantselection screen 260 includes participant information 262 for the newelectronic meeting, which in this example is the second code reviewmeeting for the Pluto project. Participant information 262 includesuser-specified meeting participants 264 that may be manually specifiedby a user, for example, from a list of contacts or one or more addressbooks. An Add Contacts control 266 allows a user to access varioussources of contacts, including organization-based contacts, personalcontacts, social media contacts, etc., to provide additional contacts.

Participant information 262 also includes suggested meeting participants268 that are determined, at least in part, with the assistance ofmeeting intelligence apparatus 102. As depicted in FIG. 2J, severalsuggested meeting participants 268 are displayed and controls, in theform of checkboxes, are provided to allow a user to individually selectsuggested meeting participants 268 to be included in the electronicmeeting. Controls 270 allow a user to save the selections of meetingparticipants via a “Save” control from controls 270, or return toelectronic meeting management screen 230 via a “Back” control fromcontrols 270. Saving selections of meeting participants via a “Save”control from controls 270 may invoke other functionality, for example,an electronic meeting application to generate and transmit meetinginvitations to the meeting participants. According to one embodiment,controls are provided to designate a meeting participant as a decisionmaker for the electronic meeting. The designation of a meetingparticipant as a decision maker may be used during the electronicmeeting to manage the electronic meeting, as described in more detailhereinafter.

Suggested Meeting Participants

Suggested meeting participants 268 may be determined based upon a widevariety of information and criteria. According to one embodiment,information for other electronic meetings, such as agendas, transcripts,meeting minutes, etc., are analyzed to determine suggested meetingparticipants 268 for the current electronic meeting. For example,attendees of the prior First Code Review meeting for the Pluto Projectmay be included in suggested meeting participants 268 for the currentelectronic meeting, i.e., the Second Code Review Meeting for the PlutoProject, based upon their attendance at the prior electronic meeting. Asanother example, other members of the same team, project, department,group, technology or business area, etc., may be identified as suggestedmeeting participants 268. Suggested meeting participants 268 may also bedetermined based upon position, title, and/or responsibilities with anorganization. For example, a team leader or decision maker for the PlutoProject may be included as one of the suggested meeting participants268. Prior participation of participants in other electronic meetingsmay also be considered, for example, based upon records of otherelectronic meetings, such as transcripts, meeting minutes, or otherpost-meeting processing, as described in more detail hereinafter. Forexample, positive participation or contributions to other electronicmeetings may be a positive factor in including a participant as asuggested meeting participants 268. Participants outside of anorganization may also be included in suggested meeting participants 268,for example, as SMEs, as described in more detail hereinafter. Suggestedmeeting participants 268 may be displayed on participant selectionscreen 260 in different ways, depending upon a particularimplementation. For example, suggested meeting participants 268 may bearranged on participant selection screen 260 based upon relevance scores(described in more detail hereinafter), the level of participation inone or more prior electronic meetings, position or role within anorganization or with respect to the electronic meeting or a subject ofthe electronic meeting.

Subject Matter Experts (SMEs)

Suggested meeting participants 268 may be determined based uponattributes of an electronic meeting. The attributes may be any type ofattributes for an electronic meeting and may include, for example, ameeting subject, meeting type, meeting location, agenda items, etc.According to one embodiment, participants determined to be highlycorrelated to the attributes for an electronic meeting are selected assuggested meeting participants. For example, suppose that the agenda forthe Second Code Review Meeting includes an agenda item pertaining toencryption. Suggested meeting participants 268 may include a suggestedmeeting participant who is a Subject Matter Expert (SME) in encryption,based upon the Second Code Review meeting including an agenda itempertaining to encryption. In the example depicted in FIG. 2J, “John E.”is identified as a SME based upon his expertise in the area ofencryption and is included in suggested meeting participants 268. Theidentification of an SME may be made based upon an analysis ofelectronic documents and other information that show that John E is aSME in encryption. Example information that may be considered includes,without limitation, resumes, academic credentials, academic papers,employee records, organization charts, presentation materials,information from other meetings, such as agendas and meeting minutes,project and product documentation, etc. SMEs may be internal or externalto an organization, depending upon a particular implementation. Forexample, some areas of technology, medicine, etc., may be very specificand have relatively few SMEs, and it is possible that some organizationsmay not have an SME available for a particular area. In thesesituations, including SMEs that are external to an organization providesadditional options for the meeting organizer.

In situations where multiple SMEs are identified for a particularsubject matter, all of the identified SMEs may be included as suggestedmeeting participants and differentiated by their respective relevancescores and additional information, which are described in more detailhereinafter. For example, a SME with prior experience on the projectthat is the subject of the electronic meeting may be given a higherrelevance score than another SME with equivalent background anexperience, but without specific prior experience on the project.Similarly, the participation level of each SME in prior electronicmeetings may be a factor in determining a relevance score. According toan embodiment, a search tool is provided to allow users to search formeeting participants. The search tool allows a user to specify one ormore attributes of desired meeting participants. Meeting intelligenceapparatus 102 performs a search and provides search results that satisfythe one or more attributes specified by the user. For example, a usermay enter, as attributes, one or more keywords associated with aparticular subject or topic, such as “encryption,” and participantsassociated with the subject “encryption” are presented to the user. Theuser may specify as search criteria other information, such as name,background, etc. The search tool may be implemented via one or morewindows, e.g., on participant selection screen 260.

Meeting Rules Templates

Suggested meeting participants 268 may be determined based upon one ormore meeting rules templates specified for an electronic meeting. Aspreviously described herein, a meeting rules template specifies a set ofrequirements and/or constraints that may be applied to any aspect of anelectronic meeting, before, during, and after an electronic meeting. Therequirements and/or constraints may pertain to meeting participants. Forexample, a meeting rules template may specify that a particular person,a person with a specified level of authority, a decision maker, or aminimum and/or a maximum number of participants must attend anelectronic meeting.

According to one embodiment, suggested meeting participants 268 includemeeting participants that are determined to satisfy one or morerequirements and/or constraints specified by one or more meeting rulestemplates designated for an electronic meeting. For example, aparticular meeting rules template may require that a Chief ExecutiveOfficer (CEO) attend a Board of Directors meeting. In this examplesuggested meeting participants 268 would include the CEO of theorganization for the electronic meeting. As another example, aparticular meeting rules template may require that corporate legalcounsel be included in a litigation settlement meeting. In this example,one of the organization's corporate legal counsel may be included insuggested meeting participants 268. As yet another example, a particularmeeting rules template may require that a representative of a HumanResources (HR) department within an organization attend an employeedisciplinary meeting. These are just a few examples of particularparticipants that are required to attend electronic meetings to satisfyone or more requirements and/or constraints specified by one or moremeeting rules templates for an electronic meeting, and embodiments arenot limited to these examples. According to one embodiment, particularparticipants that are required to satisfy requirements or constraintsfor an electronic meeting may be automatically pre-selected. Forexample, suppose that a particular member of a team is required toparticipate in an electronic meeting. The electronic meeting applicationincludes the particular member of the team in suggested meetingparticipants 268 and pre-selects the particular member. De-selecting theparticular member would cause a message to be displayed notifying theuser that the particular member is required by the meeting rulestemplate designated for the electronic meeting.

Suggested meeting participants 268 may include multiple suggestedmeeting participants for a single requirement or constraint specified bya meeting rules template, or for multiple requirements and constraintsspecified by a meeting rules template. For example, a meeting rulestemplate for a Board of Directors meeting may require that both at leasta minimum number of Board members attend the electronic meeting tosatisfy a quorum requirement for voting purposes, and also that theChairman of the Board be one of the meeting attendees. Suggested meetingparticipants 268 that are required to satisfy one or more requirementsand/or constraints specified by one or more meeting rules templates foran electronic meeting may be conspicuously identified to users. Forexample, such suggested meeting participants 268 may be displayed withan indication that they are required, and/or displayed using differentcolors, special effects, etc.

According to one embodiment, an electronic meeting application isconfigured to enforce one or more constraints or requirements specifiedby a meeting rules templates designated for an electronic meeting.Enforcement of constraints or requirements may occur at different timesduring meeting participant selection, depending upon a particularimplementation. For example, constraints may be enforced after a userhas selected meeting participants and selected the Save control 270.This may include, for example, determining whether a total number ofmeeting participants selected by a user satisfies one or moreconstraints, such as a maximum number of meeting participants, specifiedby a meeting rules template designated for the electronic meeting. Inresponse to determining that the total number of meeting participantsselected by a user exceeds a maximum number of meeting participantsspecified by the meeting rules template, the user may be notified, viathe graphical user interface, that the requirement has not beensatisfied, and provide an opportunity for the user to remedy the error,for example, by de-selecting one or more meeting participants.

Relevance Scores

According to one embodiment, a relevance score is calculated anddisplayed for each suggested meeting participant 268. The relevancescore indicates a relevance of the suggested meeting participant to theelectronic meeting, and the relevance score may be helpful to a user indeciding whether to select a particular suggested meeting participantfor inclusion in an electronic meeting. Relevance scores may beconsidered in the presentation of suggested meeting participants 268 toa user. For example, suggested meeting participants 268 may be orderedon participant selection screen 260 based upon relevance scores, e.g.,from highest (most relevant) to lowest (least relevant).

In the example depicted in FIG. 2J, the relevance scores are depicted inparentheses, i.e., “(98%)” on a scale of 0-100%, where 0% represents alowest level of relevance, or no relevance, between a particularsuggested participant and an electronic meeting, and 100% represents ahigh level of relevance, or complete relevance between a particularsuggested participant and an electronic meeting. The 0-100% scaledepicted in FIG. 2J is provided as an example implementation.Embodiments are not limited to this example and any scoring methodologymay be used.

Relevance scores may be calculated based upon a wide variety of factorsand information, such as one or more of, a meeting rules templatespecified for an electronic meeting, one or more attributes of theelectronic meeting, such as meeting type, meeting subject, meetinglocation, and agenda items, as well as any other information that mightindicate a relevance between a suggested participant and an electronicmeeting, such as participation and contributions made during priorelectronic meetings. For example, participation in a prior electronicmeeting of the same meeting type and for the same meeting subject wouldincrease the relevance score for a particular suggested meetingparticipant 268. A determination that the particular suggested meetingparticipant 268 participated in the prior electronic meeting wouldfurther increase the relevance score for the particular suggestedmeeting participant 268. This may be determined, for example, by ananalysis of records for other electronic meetings, such as meetingminutes, transcripts, etc. Information external to an organization maybe considered in determining relevance scores, for example, via theInternet. Examples of such information include, without limitation,educational background, professional credentials, work experience,published books, papers, studies and articles, other careerachievements, documents related to conferences, documents related tostandards organizations and public organizations, government documents,school curriculums, social media data, such as posts, etc. According toone embodiment, the availability of a suggested participant may bedetermined, e.g., via a participant's calendar, and indicated viaparticipant selection screen 260. For example, a visual indication maybe provided on participant selection screen 260 to indicate that aparticular participant is not available at the scheduled date and timefor an electronic meeting. This allows the meeting organizer to considerre-scheduling an electronic meeting for a meeting participant whoseparticipation is considered to be important.

Additional Information

According to one embodiment, additional information is determined anddisplayed with suggested meeting participants 268 to aid users indeciding whether to select a particular suggested meeting participantfor inclusion in an electronic meeting. The additional information mayinclude any information about the suggested participant, includinginformation used in calculating a relevance score. Examples ofadditional information include, without limitation, information about asuggested meeting participant, such as information about theireducational background, professional credentials, work experience,published books, papers, studies, articles, achievements, contributions,roles and positions within organizations, including participation inprojects and meetings, key quotes from other electronic meetings, etc.

For example, as depicted in FIG. 2J, the additional information forsuggested meeting participant “Taryn K.” specifies that this participantattended a prior electronic meeting that is relevant to the currentelectronic meeting, i.e., the First Code Review meeting for the PlutoProject. The additional information for suggested participant “Taryn K.”also specifies that this participant was a designated decision maker inthe First Code Review meeting for the Pluto Project. Both of these itemsof additional information may be helpful to a user in deciding whetherto select suggested meeting participant “Taryn K.” for inclusion in thecurrent electronic meeting, i.e., the Second Code Review Meeting for thePluto Project. These items of additional information may also havecontributed to suggested meeting participant “Taryn K.” having a veryrelevance score of 98%, i.e., based upon the facts that “Taryn K.” bothattended the First Code Review meeting for the Pluto Project and was thedesignated decision maker in that meeting.

In some situations, a large amount of additional information may beavailable for suggested meeting participants. According to oneembodiment, controls are provided on participant selection screen 260 toaccess further additional information. In FIG. 2J, an example control is“<more>” which, when selected, provides access to further additionalinformation for suggested meeting participant “John E.” The additionalinformation may be displayed, for example, in a pop-up box or overlaidwindow and may include, for example, any of the information describedabove, such as key quotes from prior meetings, etc.

V. Real-Time Processing

According to one embodiment, artificial intelligence is used to managevarious aspects of electronic meetings. For example, meetingintelligence apparatus 102 may intervene during electronic meetings toprovide any of a variety of intervention data, such as visualindications, messages in message window 224, participant information,recommendation information, and/or any other data that meetingintelligence apparatus 102 transmits during an electronic meeting.

FIG. 3 is a block diagram that depicts an arrangement for generatingintervention data. Referring to FIG. 3, meeting intelligence apparatus102 receives audio/video data 300 from node 104A. Audio/video data 300may be one or more data packets, a data stream, and/or any other form ofdata that includes audio and/or video information related to anelectronic meeting. In the example depicted in FIG. 3, audio/video data300 includes first meeting content data 302 which, in turn, includes cue304. Cue 304 may take many forms that may vary depending upon aparticular implementation. Examples of cue 304 include, withoutlimitation, one or more keywords, tones, sentiments, facialrecognitions, etc., that can be discerned from audio/video data 300.Other examples of cue 304 include whiteboard sketches and/or gesturesthat may not be part of audio/video data 300.

Meeting intelligence apparatus 102 includes cue detection logic 306,which analyzes audio/video data 300 to determine whether audio/videodata 300 includes cue 304. Cue detection logic 306 may analyzeaudio/video data 300 on a continuous basis, or on a periodic basis,depending upon a particular implementation. Meeting intelligenceapparatus 102 also includes data generation logic 308, which generatesintervention data 310 if audio/video data 300 includes cue 304. Meetingintelligence apparatus 102 transmits intervention data 310 to node 104Aduring and/or after an electronic meeting. Intervention data 310includes second meeting content data 312 that may supplement or replacefirst meeting content data 302, as described in more detail hereinafter.Intervention data may also Meeting intelligence apparatus 102 may canintervene in an electronic meeting in a wide variety of ways.Non-limiting examples include intervening to manage meeting flow, toprovide information retrieval services, and/or to supplement meetingcontent.

Meeting Management

According to one embodiment, artificial intelligence is used to assistwith managing various aspects of electronic meetings. This may include awide variety of functionality that may vary depending upon a particularimplementation. Example functionality includes, without limitation,agenda management, points of agreement, action items, new meetingrequests, and sentiment analysis. The use of speech and/or textrecognition provides a more favorable user experience by allowing usersto manage various aspects of electronic meetings using voice commandsand/or text commands.

i. Agenda Management

According to one embodiment, artificial intelligence is used to provideagenda management functionality during electronic meetings. Agendamanagement functionality may include a wide variety of functionalitythat may vary depending upon a particular implementation. Examplefunctionality includes, without limitation, enforcing time constraintsfor agenda items, changing designated amounts of time for agenda items,changing, deleting and adding agenda items, including providing missingor supplemental information for agenda items, and agenda navigation.

FIG. 4A is a block diagram that depicts an arrangement in which meetingintelligence apparatus 102 includes speech or text recognition logic 400that processes first meeting content data 302 to determine one or morecorresponding agenda topics. In the example depicted in FIG. 4A, firstmeeting content data 302 includes the speech or text statement “Grosssales are expected to be $10.8 million next quarter.” A participantassociated with node 104A may have caused first meeting content data 302to be generated by speaking, writing, typing, or displaying thestatement. Speech or text recognition logic 400 may process firstmeeting content data 302 by parsing to detect keywords that are mappedto a meeting agenda. In the present example, speech or text recognitionlogic 400 detects the keywords “next quarter.” These keywords are a cue304 for meeting intelligence apparatus 102 to generate intervention data310 that indicates a corresponding agenda topic. The intervention data310 may be used by the electronic meeting application to determine acorrespondence between a current point in an electronic meeting and ameeting agenda. This correspondence is used to provide agenda managementfunctionality, including tracking the current agenda topic. In theexample of FIG. 4A, second meeting content data 312 specifies, amongother information, the position of visual indicator 220, or a currentagenda item, using JavaScript Object Notation (JSON). Thus, one or moreelectronic meeting applications that process the JSON will displayvisual indicator 220 at the specified position in the meeting agendaduring an electronic meeting.

A determined correspondence between a current point in an electronicmeeting and a meeting agenda may be used to monitor the progress of anelectronic meeting and enforce time constraints with respect toindividual agenda items, groups of agenda items, and/or an entireelectronic meeting. This may include tracking the amount of time spenton agenda items and providing one or more indications to meetingparticipants. For example, in addition to the timer provided in agendawindow 218 (FIG. 2D), a visual and/or audible indication may be providedwhen an amount of time designated for an agenda item, group of agendaitems, or an entire electronic meeting, is nearing expiration or hasexpired. If the timer value exceeds the specified time limit, theelectronic meeting application may cause a message to be displayed inmessage window 224. The message may also be spoken by the electronicmeeting application. The message may indicate, for example, the thattime limit for the current agenda item has expired and the electronicmeeting will be progressing to the next agenda item. Additionally oralternatively, the electronic meeting application may move a visualindication to a different agenda topic. Speech and text recognition mayalso be used to ensure that all agenda items and action items areaddressed during an electronic meeting, which may include discussion,deferral, etc.

As previously described herein, agenda items may be designated asrequiring a decision, for example via one or more meeting rulestemplates, or via user-designation. According to one embodiment, anelectronic meeting application ensures that a decision is made for allagenda items requiring a decision during an electronic meeting. If auser attempts to navigate to another agenda item or action item before adecision has been made on a current agenda item, the electronic meetingapplication may display a message in message window 224, or speak themessage, indicating that the current agenda item or action item requiresa decision. This may include preventing navigation to other agenda itemsor action items until the current agenda item is addressed. A meetingowner may be permitted to override this functionality and move toanother agenda item or action item.

Speech and text recognition may be used to recognize agenda-relatedcommands, such commands to add time to agenda items and agendanavigation commands. For example, the command “add time”, followed by aspecified amount of time, adds a specified amount of time to the currentagenda item. The command “add time to” followed by an agenda item and aspecified amount of time, adds the specified amount of time to thespecified agenda item, which may be a different agenda item than thecurrent agenda item. Another example is the command “completed,” “done,”or “defer,” followed by “next agenda item,” to change the status of thecurrent agenda item and move to the next agenda item. For example, inresponse to “defer, jump to X,” the current agenda item is deferred,also referred to as “parked,” and the new current agenda item becomes X.Various other agenda navigation commands may be used, depending upon aparticular implementation.

Another example of agenda-related commands are commands to delete,change or add (new) agenda items. Agenda items may be identified by nameor other means, such as code, reference number, symbol, etc. Forexample, the command “delete” followed by a particular agenda itemcauses the particular agenda item to be deleted from the agenda. Asanother example, the command “change agenda item X to X”' causes theagenda item X to be changed to X. As yet another example, the command“add agenda item Y” or “new agenda item Y” causes a new agenda item Y tobe added to the agenda. The user may specify, for example, by speaking,writing, etc., attributes of a new agenda item, such as name, date,responsible person, etc. Agenda-related commands may also includecommands to change the order of agenda items in an agenda. For example,the command “Agenda item X, move to top” causes agenda item X to bemoved to the top of the agenda. Similarly, the command “Agenda item X,move to bottom” causes agenda item X to be moved to the bottom of theagenda.

According to one embodiment, meeting intelligence apparatus 102 providesthe capability to detect that information is missing for a new agendaitem and provide a notification to an electronic meeting application.According to this embodiment, meeting intelligence apparatus 102 detectsthat not all of the information required for a new agenda item has beenspecified. This may be performed, for example, by meeting intelligenceapparatus 102 analyzing first meeting content data 302 to detect acommand to create a new agenda item, along with some attributeinformation for the new agenda item. Referring to the prior example,meeting intelligence apparatus 102 may detect, in first meeting contentdata 302, the command, “new agenda item Y,” along with attribute datafor new agenda item Y, such as a description of the new agenda item.This command may have been spoken, written or selected by a meetingparticipant, as indicated by the first meeting content data 302.

Meeting intelligence apparatus 102 recognizes, using artificialintelligence such as artificial intelligence service 110, that someinformation for the new agenda item is missing, such as a responsibleperson. This may be determined, for example, based upon a set ofrequirements that specifies information required for agenda items. Forexample, a requirement may exist that all new agenda items, or agendaitems of this type, have an assigned responsible person. Alternatively,this may be determined based upon information specified for agenda itemsfor the same electronic meeting or other electronic meetings. Forexample, Meeting intelligence apparatus 102 may examine records forother electronic meetings to identify information provided with agendaitems for those electronic meetings and determine that most or all ofthose agenda items included a responsible person.

Once a determination has been made that a new agenda item is missingparticular information, Meeting intelligence apparatus 102 mayautomatically determine the missing information, for example, byexamining information for other agenda items for the current electronicmeeting, or for other electronic meetings. This may include identifyingagenda items that are similar to the new agenda item and then examiningthe information specified for those similar agenda items. The missinginformation may then be provided to the electronic meeting applicationfor processing. The electronic meeting application may notify themeeting participants that information is missing for the new agendaitem, for example, by displaying a message in message window 224. Theelectronic meeting application may automatically use the informationprovided by Meeting intelligence apparatus 102 to address the missinginformation and notify the meeting participants via message window 224.Alternatively, the electronic meeting application may query a meetingparticipant to provide the missing information. The information providedby Meeting intelligence apparatus 102 may be provided meetingparticipants as suggested information for completing the missinginformation.

Consider the following example. Suppose that a new agenda item iscreated without a responsible person. Meeting intelligence apparatus 102determines that a responsible person is missing for the new agenda itemand analyzes agendas for other electronic meetings to identify agendaitems that are similar to the new agenda item. Various algorithms, suchas distance algorithms, may be used to identify other similar agendaitems, e.g., other agenda items that are within a specified distance ofthe new agenda item. The responsible person designated for the otheragenda item that is most similar to the new agenda item is identifiedand provided to the electronic meeting application. In the presentexample, “Alice C.” is the responsible person for the other agenda itemthat is most similar to the new agenda item.

The electronic meeting application may automatically designate theresponsible person determined by meeting intelligence apparatus 102 forthe new agenda item and notify the meeting participants via messagewindow 224. An example message is “New agenda item X is missing aresponsible person, Alice C. has been automatically designated.” Theelectronic meeting application may provide controls to allow a meetingparticipant to confirm or change the responsible person. This may alsobe done via voice commands. Alternatively, the information provided bymeeting intelligence apparatus 102 may be displayed as a suggestion,along with controls to accept or change the suggestion. For example, theelectronic meeting application may display a message via message window224 such as, “New agenda item X is missing a responsible person.Suggestion: Alice C.” In situations where the information provided bymeeting intelligence apparatus 102 includes multiple suggestedresponsible persons, then the multiple suggested responsible persons mayall be listed, and controls provided for a user to select one of theresponsible persons. Alternatively, a user may simply speak the name ofthe responsible person to be designated for the new agenda item.

According to one embodiment, artificial intelligence may providesupplemental information for new agenda items. For example, in the priorexample in which the new agenda item is missing a responsible person andAlice C. is provided as a suggestion, the suggestion may includeadditional information about Alice C. that might be helpful to a meetingparticipant deciding whether to accept or change the suggestion. In thisexample, this may include information about Alice C, such as “New agendaitem X is missing a responsible person. Suggestion: Alice C.(responsible person on agenda items X1 and X2).” Supplementalinformation may pertain to other aspects of a new agenda item. Forexample, additional information about the subject of a new agenda itemmay be displayed.

As demonstrated by the foregoing examples and embodiments, artificialintelligence may be used to aid in various aspects of agenda managementand provide a more useful experience. Information provided by meetingintelligence apparatus 102, such as recognized commands, missing andsupplemental agenda information, etc., may be automatically provided bymeeting intelligence apparatus 102 to an electronic meeting application.Alternatively, this information may be provided in response to requestsfrom electronic meeting applications.

Changes to agenda items and new agenda items may be recorded in meetingrecords, such as meeting minutes or transcripts, and used during thecreation of other meeting agendas, as previously described herein. Theapproach described above for new agenda items may be used for existingagenda items. For example, suppose that an existing agenda item ismissing information. A user may be queried to supply the missinginformation for the existing agenda item. Alternatively, the missinginformation may be provided or suggested by meeting intelligenceapparatus 102 as previously described herein. Meeting intelligenceapparatus 102 may transmit, to an electronic meeting application, dataindicating that required information is missing for a new agenda item.In response to receiving this data, the electronic meeting applicationnotifies meeting participants, for example, via a message in messagewindow 224, an audible indication, or both, that additional informationis required for the new agenda item.

According to one embodiment, the aforementioned approaches may be usedto create new agenda items for electronic meetings that have no agenda.For example, meeting intelligence apparatus 102 may recognize a “createagenda” command and provide a notification to an electronic meetingapplication to create an agenda for the current electronic meeting.Agenda items may then be added to the agenda as previously describedherein.

ii. Points of Agreement, Action Items and New Meeting Requests

Action items are an important aspect of many electronic meetings.According to one embodiment, an electronic meeting application, inconjunction with meeting intelligence apparatus 102, is configured torecognize commands pertaining to action items for an electronic meeting.For example, meeting intelligence apparatus 102 may recognize a spokenor written command “add action item X,” and in response to this command,add action item X to a meeting record for the current electronicmeeting. Explicit commands to establish action items may bepre-specified and meeting intelligence apparatus 102 may be configuredto detect the pre-specified commands in first meeting content data 302.

According to an embodiment, meeting intelligence apparatus 102 isconfigured to recognize implicit requests for action items. Implicitrequests for action items may be recognized based upon an analysis offirst meeting content data 302. The analysis may identify words thatalthough do not constitute an explicit request for an action item,indicate a desire by one or more meeting participants to create anaction item. For example, a meeting participant may state, during acurrent electronic meeting, “Let's ask the IT department to follow-up onRF link reliability.” While this phrase may not include pre-specifiedkeywords for creating an action item, based upon the use of artificialintelligence, meeting intelligence apparatus 102 determines that thereexists an intent by one or more meeting participants is to create anaction item. In response to identifying an explicit or implicit requestfor an action item, the electronic meeting application and/or meetingintelligence apparatus 102 may automatically create a new action itemfor the electronic meeting, which may include adding the action item toan electronic record for the electronic meeting. The electronic meetingapplication may display, for example, on electronic meeting screen 212,a visual confirmation for the meeting participants indicating that theaction item was added for the current electronic meeting.

For explicit and implicit requests, the electronic meeting applicationmay display detected words or phrases and provide controls for a meetingparticipant to confirm the request for an action item. This may behelpful to avoid creating action items based upon implicit requestswhere the meeting participants did not actually intend to create anaction item. Confirmation may be provided verbally or detected viaspeech or text recognition logic 400. FIG. 2F depicts a message window224 displayed on electronic meeting screen 212 that includes a proposedaction item determined by meeting intelligence apparatus 102 analyzingfirst meeting content data 302, along with controls for confirming orcanceling the proposed action item. In this example, an explicit commandwas not used to create a new action item. Instead, meeting intelligenceapparatus 102 identified an implicit request to create a new action itemfrom the statement made by a meeting participant, “Let's ask the ITdepartment to follow-up on RF link reliability.”

There are times during some electronic meetings where certain decisions,conclusions, stipulations, or points of agreement are established by themeeting participants and it is desirable to document this information,in association with the current electronic meeting. According to oneembodiment, an electronic meeting application, in conjunction withmeeting intelligence apparatus 102, is configured to recognizedecisions, conclusions, stipulations, or points of agreement based uponexplicit commands, and in response, add the decisions, conclusions,stipulations, or points of agreement to meeting notes, minutes and/ortranscripts for the current electronic meeting. For example, meetingintelligence apparatus 102 may recognize a spoken or written command“add conclusion” or “point of agreement,” and add a following conclusionstatement or point of agreement to a meeting record for the currentelectronic meeting. In addition, the electronic meeting application maydisplay, for example on electronic meeting screen 212, a visualconfirmation for the meeting participants indicating that the decision,conclusion, stipulation, or point of agreement was added for the currentelectronic meeting. Explicit commands to establish decisions,conclusions, stipulations, or points of agreement may be pre-specifiedand meeting intelligence apparatus 102 may be configured to detect thepre-specified commands in first meeting content data 302.

According to an embodiment, meeting intelligence apparatus 102 isconfigured to recognize implicit decisions, conclusions, stipulations,or points of agreement and add corresponding information to a meetingrecord for the electronic meeting. Implicit decisions, conclusions,stipulations, or points of agreement may be recognized in first meetingcontent data 302 based upon words that, based upon an artificialintelligence analysis, indicate a desire by one or more meetingparticipants to establish a decision, conclusion, stipulation, or pointof agreement, even though a pre-specified command was used by a meetingparticipant. For example, a meeting participant may state, “It appearsthat we have reached a consensus on this point.” Based upon the use ofartificial intelligence, meeting intelligence apparatus 102 determinesthat the intent of one or more meeting participants was to establish aconclusion or point of agreement, without using a pre-specified commandfor doing so. As previously described herein with respect to actionitems, message window 224 may be used to display a possible decision,conclusion, stipulation, or point of agreement, along with controls forconfirming or canceling. Message window 224 may also be used to displaydetected explicit or implicit requests to create a new meeting, alongwith controls for confirming or canceling.

i. Sentiment Analysis

As previously described herein, new meetings may be created based uponthe results of sentiment analysis that indicates that a new meeting maybe appropriate or helpful in a particular situation, for example ifsignificant disagreement and/or hostility exists between participants ina meeting. The results of sentiment analysis may also be used toinitiate other actions to address the detected sentiment, e.g., todiffuse detected hostility between meeting participants. For example, aproposal to create an action item may be generated and displayed inmessage window 224, e.g., “Create action item for current topic?” Asanother example, a proposal to create a follow-up meeting may begenerated and displayed in message window 224, e.g., “Create follow-upmeeting for current topic?” As yet another example, a proposal to deferthe current agenda item and move to the next agenda item may bedisplayed in message window 224, such as “Defer current agenda item andmove to next agenda item?” This allows a meeting participant, such asthe meeting owner to diffuse the detected hostility by stoppingdiscussion on the current agenda item and moving to another agenda item.Furthermore, the suggestion of the alternative actions by meetingintelligence apparatus 102 may depersonalize the intervention because itis done by meeting intelligence apparatus 102 in combination with theelectronic meeting application, rather than a meeting participant. Thismay provide a more favorable user experience.

FIG. 4B is a block diagram that depicts an arrangement for performingsentiment analysis with respect to an ongoing discussion 402. Referringto FIG. 4B, meeting intelligence apparatus 102 includes sentimentanalysis logic 404 that performs sentiment analysis on first meetingcontent data 302 related to ongoing discussion 402. For example, meetingintelligence apparatus 102 may detect an angry tone or sentiment that isa cue 304 for meeting intelligence apparatus 102 to generateintervention data 310 indicating that another electronic meeting hasbeen automatically scheduled for continuing ongoing discussion 402. Inthe example of FIG. 4B, second meeting content data 312 includes JSONthat can be used by an electronic meeting application to make decisionsabout a current electronic meeting.

Information Retrieval

According to one embodiment, an electronic meeting application, inconjunction with meeting intelligence apparatus 102, providesinformation retrieval services for electronic meetings. This capabilityallows a user to formulate and submit natural language queries forprocessing, and view the results of processing the natural languagequeries during an electronic meeting. This provides a more favorableuser experience than requiring users to be able to formulate queries inmore formal formats or a computer language, such as Structured QueryLanguage (SQL).

FIG. 4C is a block diagram that depicts an arrangement for retrievingrequested information using natural language queries. A user specifies anatural language query to an electronic meeting application executing,for example, on node 106A. For example, FIG. 2K depicts a naturallanguage query via query window 272 of electronic meeting screen 212that allows a user to enter a natural language query. The electronicmeeting application generates and transmits a natural language request406 to meeting intelligence apparatus 102. Natural language request 406may be included in first meeting content data 302, or may be transmittedseparately. In the example depicted in FIG. 4C, natural language request406 includes the question “Where did we leave off at the last meeting?”Note that natural language request 406 may include a question, astatement, a command, or any other type of request for information.

Speech or text recognition logic 400 parses and interprets first meetingcontent data 302 to detect natural language request 406, which is a cue304 for meeting intelligence apparatus 102 to generate intervention data310 to be sent to at least node 104A during an electronic meeting. Forexample, speech or text recognition logic 400, alone or in combinationwith sentiment analysis logic 404, may detect inflected speech and/orkeywords indicative of an information request, such as “who”, “what”,“when”, “where”, “why”, or “how”. Meeting intelligence apparatus 102 caninterpret these and other keywords as commands to perform requestedfunctions, such as data retrieval.

In the example of FIG. 4C, meeting intelligence apparatus 102 mayinterpret the question as a command to search and analyze prior meetingdata to determine an answer to the question. Determining the answer tothe question may involve analyzing meeting content data related to anongoing meeting and/or a prior meeting, thereby increasing the relevanceof the answer to the question. For example, the question “Where did weleave off at the last meeting?” may be analyzed using contextual data(e.g., metadata) from the current meeting, such as the identities ofparticipants 108A-N, the topic of the current discussion, etc. Meetingintelligence apparatus 102 may search the meeting data repository forinformation that most closely matches the contextual data from thecurrent meeting. For example, meeting intelligence apparatus 102 maysearch the meeting data repository for any prior meetings that includedsome or all of the participants 108A-N of the current meeting and rankthe results. Meeting intelligence apparatus 102 may then determine thatthe “last meeting” refers to the top result and may search for the lastagenda topic in the prior meeting that corresponds to the top result.

Intervention data 310 that is generated in response to natural languagerequest 406 includes stored information 410 that meeting intelligenceapparatus 102 retrieves in response to natural language request 406.Meeting intelligence apparatus 102 includes data retrieval logic 408,which performs a search for stored information 410 that is responsive tonatural language request 406. For example, data retrieval logic 408 maysearch a meeting data repository and/or external data sources, such aswebsites on the Internet. In the example of FIG. 4C, meetingintelligence apparatus 102 generates second meeting content data 312that includes stored information 410 retrieved from a meeting datarepository. The stored information 410 includes the answer to thequestion about a different meeting.

In an embodiment, meeting intelligence apparatus 102 processes naturallanguage request 406 and researches a particular topic or otherwisesearch for information that is unrelated to a particular meeting. Forexample, natural language request 406 may be the statement “We need tofigure out how to get source code from the app.” In response, meetingintelligence apparatus 102 may retrieve information from variouswebsites that address natural language request 406. As shall bedescribed in greater detail hereafter, this can be a particularly usefulfeature for participants 108A-N who wish to collaborate, during anelectronic meeting, to create a presentation, a report, or any otherdocument. Results of processing natural language queries are displayedby the electronic meeting application, for example, via message window224.

Meeting Content Supplementation

According to one embodiment, an electronic meeting application, inconjunction with meeting intelligence apparatus 102, provides contentsupplementation services for electronic meetings. These services providesupplemental information for electronic meetings, including additionalinformation about agenda items, electronic meeting content, and meetingparticipants, and translation information.

According to one embodiment, meeting intelligence apparatus 102 analyzesagenda items and electronic meeting content and provides supplementalinformation that is displayed by an electronic meeting application, forexample, via message window 224. The supplemental information maypertain to one or more agenda items displayed in agenda window 218 orcontent 214 displayed in agenda window 213. The supplemental informationmay be obtained by identifying one or subjects or topics in agenda itemsor content, and then performing searches on those subjects or topics.The searches may be performed as a background task to a currentelectronic meeting and may be automatically initiated and then re-runwhen changes to agenda items or content occur. Results of the searchesmay be displayed in message window 224 and/or one or more other searchresults windows. This may include specialized search results windows,for example, a search results window for search results pertaining toagenda items and a search results widow for search results pertaining toelectronic meeting content, such as an ongoing meeting or conversation.The information in the search results windows may be continuouslyupdated to reflect changes in the agenda and the content of anelectronic meeting. For example, as an electronic meeting is conducted,first meeting content data 302 is provided to and processed by meetingintelligence apparatus 102 to identify topics or subjects to besearched, the identified topics or subjects are searched and searchresults provided to and displayed by an electronic meeting application.As the electronic meeting switches from topic to topic, the searchresults displayed by the electronic meeting application are dynamicallyupdated. This allows meeting participants to have access to supplementalinformation for an electronic meeting. According to one embodiment, dataitems in search results are selectable, via a pointing device or voicecommands, and when a data item is selected, a subsequent search isconducted for the selected data item and the results displayed. Thisallows meeting participants to seamlessly acquire additional informationabout topics or subjects discussed during an electronic meeting.

According to an embodiment, the supplemental information includesinformation about one or more meeting participants. For example, meetingintelligence apparatus 102 may analyze first meeting content data 302 toidentify one or more meeting participants, and then transmit participantidentification data one or more electronic meeting applications.

FIG. 4D is a block diagram that depicts an arrangement for supplementingmeeting content with participant identification data. Referring to FIG.4D, meeting intelligence apparatus 102 includes voice or facerecognition logic 412, which performs voice or face recognition on firstmeeting content data 302 to detect a voice or a face. The voice or faceis a cue 304 for meeting intelligence apparatus 102 to generateintervention data 310 to be sent to at least node 104A during anelectronic meeting. In response to detecting the cue 304, meetingintelligence apparatus 102 determines one or more participants 108A-Nand generates participant identification data that identifies the one ormore participants 108A-N. Meeting intelligence apparatus 102 generatesand transmits second meeting content data 312 that includes theparticipant identification data. When processed at one or more nodes104A-N, second meeting content data 312 causes the participantidentification data to be presented at the one or more nodes 104A-N.

According to one embodiment, the supplemental information includestranslation information. Meeting intelligence apparatus 102 may cause alanguage translation or format conversion of first meeting content data302 to be presented at one or more nodes 104A-N. In an embodiment,meeting intelligence apparatus 102 performs speech or text recognitionon first meeting content data 302 to detect a particular language, whichmay be a cue 304 for meeting intelligence apparatus 102 to generatesecond meeting content data 312 that includes a translation of firstmeeting content data 302 into a different language. For example, meetingintelligence apparatus 102 may translate English content into Japanesecontent. Second meeting content data 312 may replace or supplement firstmeeting content data 302. For example, second meeting content data 312may cause Japanese dubbing of first meeting content data 302, may causeJapanese subtitles to be added to first meeting content data 302, or forJapanese content to be provided separate from the original Englishcontent. Embodiments are described herein the context of translationfrom English to Japanese for explanation purposes only, and translationsmay be provided from any source language into any target language.According to one embodiment, different translations may be provided fordifferent nodes 104A-N participating in an electronic meeting. Forexample, suppose that an electronic meeting is initiated at a first nodein a first language, and that second and third nodes are participatingin the electronic meeting. At the second node, meeting content may betranslated from the first language to a second language, and at thethird node, meeting content may be translated from the first languageand/or the second language to a third language. The translations may beprovided in real-time, for example, via message window 224 and/or viaone or more other windows. Users at each participating node may specifya language for their node, for example via meeting controls 222, whichmay be used as a default language for that node. Users may also specifya preferred language in their user profile, or in association with theiruser credentials, to allow an electronic meeting application toautomatically default to the preferred language for a meetingparticipant. This translation approach allows multiple participants toparticipate in an electronic meeting with other participants in multipleother languages, greatly enhancing the user experience and theeffectiveness of electronic meetings.

In an embodiment, meeting intelligence apparatus 102 can detect inputfrom an input/output mechanism, and the input may be a cue 304 formeeting intelligence apparatus 102 to convert the input into a differentformat. For example, the input/output mechanism may be an electronicwhiteboard that receives as input first meeting content data 302 in theform of handwritten notes or hand-drawn illustrations. Based on opticalcharacter recognition (OCR), vector graphics, and/or any other dataconversion tool, meeting intelligence apparatus 102 may convert firstmeeting content data 302 into second meeting content data 312 in theform of machine-lettering or a machine-drawn image. When processed atone or more nodes 104A-N, second meeting content data 312 may cause themachine-lettering or the machine-drawn image to be provided as output onthe electronic whiteboard.

Cue detection logic 306, data generation logic 308, speech or textrecognition logic 400, sentiment analysis logic 404, data retrievallogic 408, and voice or face recognition logic 412 may be implemented onmeeting intelligence apparatus 102 as part of artificial intelligenceservice 110, or separate from meeting intelligence service 110. Inaddition, these elements may be implemented by computer hardware,computer software, or any combination of computer hardware and computersoftware.

Meeting Content Metadata Generation

FIG. 4A-D each depict second meeting content data 312 that includes avariety of meeting content metadata. Meeting intelligence apparatus 102generates the meeting content metadata based on internal and/or externalinformation. Internal information includes information readilyaccessible to meeting intelligence apparatus 102 even in the absence ofa network connection. For example, if meeting intelligence apparatus 102is a computer, the system date and time are internal information. Incontrast, external information includes information accessible tomeeting intelligence apparatus 102 via a network connection. Forexample, information retrieved from external data sources are externalinformation.

FIG. 4A-D each depict sending meeting content metadata to one or morenodes 104A-N during an electronic meeting. However, some meeting contentmetadata may remain un-transmitted throughout the duration of theelectronic meeting. For example, some meeting content metadata mayremain stored in meeting intelligence apparatus 102 for an internal use,such as generating a report. As shall be described in greater detail inFIG. 6C, a notable example of such meeting content metadata is a labelthat identifies a key meeting point, such as an action item, a task, adeadline, etc.

VI. Post-Meeting Processing

According to one embodiment, an electronic meeting process, inconjunction with meeting intelligence apparatus 102, may provide a widevariety of post-meeting processing services, based upon analyzingmeeting content. As used herein, the term “meeting content” refers toany information pertaining to one or more electronic meetings. Examplesof meeting content include, without limitation, meeting scheduling andagenda information, meeting participant information, content sharedduring an electronic meeting, and all actions and functions performedduring one or more electronic meetings.

Meeting content may be analyzed at any time relative to an electronicmeeting, i.e., before, during or after an electronic meeting, as soon asat least some meeting content is available. Certain types of processing,such as participant analysis described in more detail hereinafter, maybenefit from being able to process meeting data from one or morecompleted electronic meetings. For example, after an electronic meetingends, meeting intelligence apparatus 102 may analyze stored meetingcontent data and generate reports based on analyzed meeting contentdata. Alternatively, meeting intelligence apparatus 102 may analyzemeeting content data during an electronic meeting and may generate,after the electronic meeting ends, a report based on analyzed meetingcontent data. Reports may include any type and amount of data, includingany number of documents, a meeting agenda, a meeting summary, a meetingtranscript, a meeting participant analysis, a slideshow presentation,etc. As previously described herein, post meeting processing results maybe used for other electronic meetings. For example, post meetingprocessing results may be used to determine suggested meetingparticipants for other electronic meetings. This may be repeated toimprove the quality of suggested meeting participants over time.

FIG. 5 is a block diagram that depicts an arrangement for processingmeeting content and generating a report. Referring to FIG. 5, meetingintelligence apparatus 102 receives, from node 104A, audio/video data300 that includes first meeting content data 302. Meeting intelligenceapparatus 102 includes data extraction logic 500, metadata generationlogic 502, and report generation logic 506. Data extraction logic 500causes first meeting content data 302 to be extracted from audio/videodata 300. Meeting intelligence apparatus 102 analyzes first meetingcontent data 302 and uses metadata generation logic 502 to generatemeeting content metadata 504. Report generation logic 506 causes meetingcontent metadata 504 to be included in report 508. This processing mayoccur on a continuous basis, or in response to a request to perform theprocessing.

Meeting intelligence apparatus 102 may perform various functions basedupon report 508. For example, meeting intelligence apparatus 102 maystore report 508 in a meeting data repository or provide report 508 toone or more nodes 104A-N associated with participants 108A-N of anelectronic meeting. Thus, meeting intelligence apparatus 102 maygenerate report 508 in an offline mode and/or an online mode. Electronicmeeting applications may also process and store report 508, as well asother data generated in response to an analysis of meeting content.

Meeting Content Analysis

In an embodiment, meeting intelligence apparatus 102 generates meetingcontent metadata 504 during an electronic meeting. For example, datageneration logic 308 may include metadata generation logic 502, andsecond meeting content data 312 may include meeting content metadata504. FIG. 6A-C depict examples of meeting content metadata 504 that canbe generated during an electronic meeting.

FIG. 6A is a block diagram that depicts an arrangement for generatingmeeting content metadata 504 that includes participant identificationdata 602. Referring to FIG. 6A, data extraction logic 500 extracts andprovides first meeting content data 302 to metadata generation logic502. In the example of FIG. 6A, metadata generation logic 502 includesvoice or face recognition logic 412, which performs voice or facerecognition on first meeting content data 302 to identify one or moreparticipants 108A-N in an electronic meeting. Metadata generation logic502 generates meeting content metadata 504 that includes participantidentification data 602 for the one or more participants 108A-N.Metadata generation logic 502 provides meeting content metadata 504 toreport generation logic 506.

FIG. 6B is a block diagram that depicts an arrangement for generatingmeeting content metadata 504 that includes a sentiment detected in firstmeeting content data 302. Referring to FIG. 6B, data extraction logic500 extracts first meeting content data 302 that includes the statement“Not necessarily.” Metadata generation logic 502 includes sentimentanalysis logic 404, which performs sentiment analysis on first meetingcontent data 302 to determine a sentiment 604 of a participant in anelectronic meeting. Sentiment analysis may be performed using a widevariety of sentiment processing algorithms that may vary depending upona particular implementation. Metadata generation logic 502 generatesmeeting content metadata 504 that includes sentiment 604. In the exampleof FIG. 6B, meeting content metadata 504 also includes participantidentification data 602 and information related to providing atranslation of first meeting content data 302. Thus, metadata generationlogic 502 may include a combination of sentiment analysis logic 404,voice or face recognition logic 412, and speech or text recognitionlogic 400.

FIG. 6C is a block diagram that depicts an arrangement for generatingmeeting content metadata 504 that includes a label to identify a key,i.e., important, meeting point. Referring to FIG. 6C, first meetingcontent data 302 includes the statement “Action item create schedule byTuesday”. Metadata generation logic 502 includes speech or textrecognition logic 400, which performs speech or text recognition onfirst meeting content data 302 to recognize one or more keywords 606 infirst meeting content data 302. One or more keywords 606 may indicate atask 608 to be completed after an electronic meeting. For example, theone or more keywords 606 may include a voice or text command to performa particular task. In the example of FIG. 6C, the one or more keywords606 are the label “Action item” followed by the command “create scheduleby Tuesday.” Metadata generation logic 502 generates meeting contentmetadata 504 that includes the one or more keywords 606 and/or task 608.

Meeting intelligence apparatus 102 may generate meeting content metadata504 based on internal and/or external information, such as geolocationinformation or a meeting room availability schedule. In each of FIG.6A-C, report generation logic 506 includes meeting content metadata 504in report 508. FIG. 7A-B depict examples of report 508. Referring toFIG. 7A-B, meeting intelligence apparatus 102 provides report 508 via aweb-based participant interface. Meeting intelligence apparatus 102 maysend report 508 to one or more nodes 104A-N at any of a number of times,such as upon demand, upon detecting a network connection, automaticallyafter each electronic meeting, etc.

Meeting Results

Post-meeting processing may be performed in accordance with requirementsand constraints specified for an electronic meeting, e.g., via a meetingrules template. For example, a meeting rules template may specifyparticular types of documentation to be created for an electronicmeeting, such as meeting minutes, transcriptions, summaries,translations, etc. The types and form of documentation may be requiredto satisfy various organizational, governmental, judicial, andcompliance requirements. The meeting rules template may also specify howthe documentation is to be handled, e.g., stored in a particularlocation, distributed to meeting participants and others, etc. Forexample, a meeting rules template may specify that meeting results areto be stored on a particular file server and emailed to all of themeeting participants.

FIG. 7A depicts an example meeting reports screen 700 that providesaccess to meeting results data for a plurality of electronic meetings.Meeting reports screen 700 may be accessed, for example, by a userselecting the “Meeting Reports” control from controls 202 on electronicmeeting screen 200 (FIG. 2A). Meeting reports screen allows a user toselect a particular electronic meeting and view meeting results for theselected electronic meeting.

FIG. 7B depicts an example meeting results screen 710 for the CometDesign Review Meeting displayed, for example, in response to a userusing a pointing device, such as a mouse, to select the “Comet DesignReview Meeting” from meeting reports screen 700. Meeting results screen710 displays meeting results for a selected electronic meeting. In theexample depicted in FIG. 7A, the meeting results information ispresented in the form of a meeting summary “dashboard,” that includesmany of the meeting content metadata 504 depicted in FIG. 6A-C, butembodiments are not limited to this example and are applicable to anytype of meeting results.

Meeting results, including a meeting summary, may include explicit dataand/or implicit data. Explicit data includes information such as detailsabout the electronic meeting (name, subject, date, location, etc.),meeting participants, meeting content data, such as documents, images,and/or any other data shared during an electronic meeting. In theexample of FIG. 7B, explicit data includes the meeting agenda, a list ofaction items, and a list of documents. In this example, the meetingagenda is accessed via a link, but one or more portions of an electronicmeeting agenda, including an entire agenda, may be included in meetingresults screen 710.

Implicit data includes meeting content metadata 504, such asidentifiers, translations, and/or any other data generated by meetingintelligence apparatus 102 after analyzing an electronic meeting. In theexample depicted in FIG. 7B, meeting results screen 710 includes a linkto a transcript of the electronic meeting, and a user may select alanguage for the transcript, irrespective of the language in which theelectronic meeting was conducted. Multiple transcripts in differentlanguages may be created. For example, for an electronic meetingconducted in English, both English and Japanese transcripts may begenerated. Links are also provided to individual reports related to eachparticipant which, as described in more detail hereinafter, may includevarious participant metrics.

In the example of FIG. 7B, the meeting results screen 710 also includeslinks to other reports, such as the meeting agenda, action items, andvarious documents generated during the electronic meeting. In thisexample, the electronic documents that were generated during theelectronic meeting include “Whiteboard Notes” and “Draft Schedule,” andthese electronic documents were generated based on one or moreinput/output mechanisms. For example, the one or more input/outputmechanisms may include an electronic whiteboard. Meeting intelligenceapparatus 102 may convert handwritten notes or hand-drawn illustrationsreceived as input on the electronic whiteboard into a machine-letteredor a machine-drawn image based on optical character recognition (OCR),vector graphics, and/or any other data conversion tool. For example,meeting intelligence apparatus 102 may perform OCR on handwritten notesto generate metadata that indicates which letters were detected. Themetadata may then be used to generate the detected letters in aparticular font or any other machine-lettering format.

According to an embodiment, post-meeting processing includes performingvarious statistical analyses, for example, to determine meetingefficiency. In the example of FIG. 7B, meeting results screen 710includes a pie chart 712 that depicts an amount of time spent on each oftwo agenda items during an electronic meeting. In the present example,pie chart 712, along with the accompanying key, indicates that 25% ofthe meeting time was spent discussing the first agenda item, “Createschedule by Tuesday” and 75% of the meeting time was spent discussingthe second agenda item, “Get feedback from Marketing.” Similarapproaches may be used for other aspects of an electronic meeting, suchas action items, and information may be arranged by subject or topic.Embodiments are not limited to this example using pie charts, and anymethod of presenting statistical information on a computer userinterface may be used.

Meeting results screen 710 also includes a statistic in the form of ameeting efficiency metric 714. In the example depicted in FIG. 7B,meeting efficiency is represented by the integer number “86” that isdepicted on a scale from low to high, and relative to the meetingefficiency for all electronic meetings. A low efficiency may correspondto a meeting efficiency of 0, while high efficiency may correspond to ameeting efficiency of 100. Meeting efficiency may be calculated basedupon a wide variety of criteria that may vary depending upon aparticular implementation. Example criteria include, without limitation,the actual total amount of actual time required for an electronicmeeting relative to a scheduled amount of time for the electronicmeeting, an amount of time added to agenda items during the electronicmeeting, an average amount of time per agenda item, a percentage ofagenda items that were completed during the electronic meeting, apercentage of action items that were completed during the electronicmeeting, etc. Meeting efficiency may be determined based upon anycombination of these example criteria, or other criteria, and criteriamay be weighted, depending upon a particular implementation. Meetingefficiency is just one example of statistical-based meeting results thatmay be presented to users, and other statistics may be used. A “Back”control is provided which, when selected by a user, returns control tomeeting reports screen 700 of FIG. 7A.

In the prior examples, meeting results were determined and displayed fora particular electronic meeting selected by a user. According to oneembodiment, aggregate meeting results are determined across multipleelectronic meetings. This allows a user to view overall meetingstatistics across multiple meetings with the organization. Examples ofmeeting statistics include, without limitation, meeting efficiency,average participant participation rate, action item completion rate,sentiment scores, average meeting time, average meeting time overrun,etc. Controls may allow a user to specify the electronic meetings forwhich statistics are to be determined and displayed. For example,controls may allow a user to select all electronic meetings with anorganization, or select electronic meetings by division, group,department (sales & marketing, accounting, engineering, testing, etc.),project, team, user, meeting topic, meeting objective, etc. Thestatistical results for the selected entity may be displayed in anelectronic meeting “dashboard” that allows a user to quickly and easilyview the performance of multiple electronic meetings. The statisticalresults displayed on the electronic meeting “dashboard” may bedynamically updated as the user selects a different division, group,department (sales & marketing, accounting, engineering, testing, etc.),project, team, user, etc. For example, an executive within anorganization may quickly and easily view the efficiency of electronicmeetings conducted across each entity within the organization, which insome situations allows improvements to be made to how electronicmeetings are conducted within the organization.

Participant Analysis

According to one embodiment, post-meeting processing includesparticipant analysis, which may include determining one or morestatistics that are indicative of one or more attributes of theparticipation by meeting participants, such as the quality and/orquantity of participation. FIG. 7C depicts an example participantanalysis report 720 that may be accessed by selecting a particularmeeting participant from meeting results screen 710 in FIG. 7B, i.e.,“John C.,” “Sue K.,” or “Mike B.” In the present example, it is presumedthat a user selected the meeting participant “Sue K.” and participantanalysis report 720 depicts a meeting participant profile for meetingparticipant “Sue K.”

Controls, such as “+” and “−” allow a user to view the meetingparticipant's profile for a particular electronic meeting. In theexample depicted in FIG. 7C, a user has selected the “+” controladjacent the “Comet Design Review Meeting” to view Sue K's meetingparticipant profile for this electronic meeting. As depicted in FIG. 7C,participant analysis report 720 depicts various participant metricsincluding a total participation time during the electronic meeting, aparticipation time for each agenda item discussed during the electronicmeeting, a participation index, an action item completion rate, a role,and sentiment analysis results. The sentiment analysis results may varydepending upon the type of sentiment analysis used. For example,sentiment analysis results may indicate a polarity sentiment result,i.e., positive, negative or neutral. As another example, sentimentanalysis results may include a sentiment score for a plurality ofsentiments. In the example depicted in FIG. 7C, a sentiment score, on ascale of 0-100, is provided for each of six sentiments, where a score of0 represents little or no detected presence of the correspondingsentiment, and a score of 100 represents a high presence of thecorresponding sentiment. Sentiment analysis results may include otherinformation, such as times during an electronic meeting where particularsentiments were detected. Multiple times may be indicated for particularsentiments that were detected at different times during an electronicmeeting.

The participation index may be any measure, weighted or otherwise, ofany aspect of the selected participant's contribution to the meeting.For example, “63/100” may indicate a proportion of the total meetingtime during which the selected participant spoke. The action itemcompletion rate indicates the past success of a meeting participant incompleting action items assigned to the meeting participant. The actionitem completion rate may be calculated in a wide variety of ways thatmay vary depending upon a particular implementation. For example, theaction item completion rate depicted in FIG. 7C indicates the number ofactions items that are completed by a particular meeting participantrelative to the total number of action items assigned to the particularparticipant. In the present example, meeting participant “Sue K.”completed one action item out of a total of two action items. The actionitems may or may not have been assigned specifically to meetingparticipant “Sue K.”

The role associated with the selected participant may indicate any of anumber of categories that describe the selected participant relative tothe current meeting and/or within a particular entity (e.g., avice-president of a corporation). For example, “Active Presenter” mayindicate that the selected participant did not merely respond to otherparticipants, but also provided many of the topics for discussion. Otherstatistics may be determined for participants and vary depending upon aparticular implementation.

Statistics for meeting participants may be determined based upon anynumber of electronic meetings. For example, statistics for meetingparticipants may be calculated based upon a single electronic meeting,or may be based upon aggregated data for multiple electronic meetings.Some statistics, such as the action item completion rate, may be moreuseful when determined over a large sample of electronic meetings,because the number of action items completed and assigned to anyindividual meeting participant in a single electronic meeting may besmall.

According to one embodiment, participant analysis report 720 includesaggregate statistics 722 for meeting participants over multipleelectronic meetings. In this example, aggregate statistics 712 formeeting participant “Sue K.” are based upon 12 electronic meetings andinclude a participation index, an action item completion rate, andsentiment scores. The sentiment scores are determined for each of aplurality of sentiments over 12 electronic meetings and in this exampleindicate that meeting participant “Sue K.” was often cheerful, rarelyconfused, often defensive and argumentative, sometimes pensive and notoften confident.

According to one embodiment, aggregate statistics may be displayed formultiple meeting participants and sorted. For example, in FIG. 7C,participant analysis report 720 includes aggregate statistics for allparticipants 724, ranked by participation rate and by action itemcompletion rate. This information may be useful in a wide variety ofcontexts. For example, aggregate statistics may be useful to meetingorganizers when planning meetings, to meeting owners during meetingswhen, for example, assigning action items, and for performanceevaluation purposes. Aggregate statistics may also be used by meetingintelligence apparatus 102 to determine suggested meeting participants,as previously described herein. Embodiments are not limited to theexample aggregate statistics depicted in FIG. 7B and any aggregatestatistics may be used. According to one embodiment, aggregatestatistics may be determined and displayed for logical entities, such asprojects, departments, groups, divisions, etc., within an organization.This allows users to view meeting participants performance on aper-logical entity basis. For example, a user may view meetingperformance statistics for meeting participants of a particular project,and compare those statistics to meeting performance statistics formeeting participants of other projects within an organization. Thisgreatly increases the usefulness of meeting performance statistics tomanagement or other interested personnel. A “Back” control is providedwhich, when selected by a user, returns control to meeting resultsscreen 710 of FIG. 7B.

According to one embodiment, translation services may be applied tomeeting results and participant analysis. For example, meeting resultsscreen 710 may include a language selection control that when selected,displays the contents of meeting results screen 710 in the selectedlanguage. The same approach may be applied to the electronic meeting“dashboard” previously described herein. Similarly, participant analysisreport 720 may include a language selection control that when selected,displays the contents of participant analysis report 720 in the selectedlanguage.

D. Action Item Management

According to one embodiment, post-meeting processing includes actionitem management, which may include displaying action items for one ormore electronic meetings and information for each action item, such as adescription, responsible person, due date, current status, etc. Actionitems may be viewed on a per meeting basis, or across multiple meetings.For example, action items for an entire organization, department, team,project, user, due date, status, etc., may be viewed.

VII. Process Overview

FIGS. 8 and 9 are flow diagrams that depict various processes that canbe performed by meeting intelligence apparatus 102. In an embodiment,FIG. 8 depicts a process that is performed with a network connectionduring an electronic meeting. In an embodiment, FIG. 9 depicts a processthat can be performed, at least partially, with or without a networkconnection.

Generating Intervention Data

FIG. 8 is a flow diagram 800 that depicts an approach for generatingintervention data 310. At block 802, a meeting intelligence apparatus102 receives meeting content data for an electronic meeting thatincludes a plurality of participants 108A-N. The meeting content datamay include, for example, audio/video data 300 that includes firstmeeting content data 302 for the electronic meeting. For example,meeting intelligence apparatus 102 may receive a videoconference streamfrom a Ricoh UCS P3500 associated with a meeting participant Alice, whois making an offer to a meeting participant Bob during the electronicmeeting. The meeting content data may include other data besides theactual audio/video data for the electronic meeting. For example, themeeting content data may include supporting documentation for theelectronic meeting, as well as any type of information pertaining toother electronic meetings, as well as any other type of information.

At block 804, the meeting intelligence apparatus 102 determines that themeeting content data, e.g., audio/video data 300, includes a cue 304 forthe meeting intelligence apparatus 102 to intervene in the electronicmeeting. The meeting intelligence apparatus 102 may make thisdetermination based on performing any of a number of analyses on themeeting content data, such as speech or text recognition, voice or facerecognition, sentiment analysis, etc. For example, meeting intelligenceapparatus 102 may extract and analyze first meeting content data 302 todetect poor eye contact by Alice. The poor eye contact may be a cue 304for meeting intelligence apparatus 102 to respond by sending arecommendation to Bob.

At block 806, the meeting intelligence apparatus 102 generatesintervention data 310 in response to detecting the cue 304. Theintervention data 310 includes second meeting content data 312 that isdifferent from the first meeting content data 302. For example, meetingintelligence apparatus 102 may generate a recommendation that advisesBob to make a counteroffer.

At block 808, the meeting intelligence apparatus 102 sends theintervention data 310 to one or more nodes 104A-N during the electronicmeeting. The one or more nodes 104A-N are associated with at least oneparticipant of the plurality of participants 108A-N. For example,meeting intelligence apparatus 102 may send the recommendation to Boband withhold the recommendation from Alice.

Generating Reports

FIG. 9 is a flow diagram 900 that depicts an approach for generating areport 508. At block 902, a meeting intelligence apparatus 102 receivesmeeting content data for an electronic meeting that includes a pluralityof participants 108A-N. For example, meeting intelligence apparatus 102may receive an audioconference data packet from Charlie's smartphone,which is executing the Ricoh UCS app.

At block 904, meeting intelligence apparatus 102 analyzes the meetingcontent data. For example, meeting intelligence apparatus 102 mayanalyze audio/video data 300 for an electronic meeting and extractparticular meeting content data from the audio/video data 300. This mayinclude, for example, stripping out header data and analyzing thepayload of audioconference data packets. Analyzing the payload mayinvolve performing speech or text recognition, sentiment analysis, voiceor face recognition, etc.

At block 904, meeting intelligence apparatus 102 generates meetingcontent metadata 504 based on analyzing the meeting content data. Forexample, meeting intelligence apparatus 102 may perform voicerecognition on the meeting content data to identify Charlie as theperson presenting at the electronic meeting. meeting intelligenceapparatus 102 may generate JSON that includes “speaker: Charlie” amongthe name-value pairs.

At block 906, the meeting intelligence apparatus 102 includes at leastpart of the meeting content metadata 504 in a report 508 of theelectronic meeting. For example, meeting intelligence apparatus 102 maygenerate a “Meeting Summary” report that includes “Charlie” among theparticipants 108A-N of the electronic meeting.

VIII. Translation and Transcription

As previously described herein, translation and transcription servicesmay be used in conjunction with electronic meetings, for example, tosupplement meeting content and to generate post meeting reports. A widevariety of translation and transcription services may be used andembodiments are not limited to any particular translation ortranscription services. As used herein, the term“translation/transcription” refers to translation, transcription, orboth translation and transcription, where translation refers toconverting text in one language to text in one or more other languages,or converting human speech from one language to one or more otherlanguages, and transcription refers to converting human speech to text.

One of the problems with conventional translation/transcription servicesis that they typically provide varying results, i.e., different levelsof accuracy, based upon factors such as language, context, speaker,length of phrases, etc. For example, a particulartranslation/transcription service may provide excellent results whentranslating from language A to language B, but inferior results whentranslating from language A to language C. Similarly, a particulartranslation/transcription service may provide excellent results whentranslating/transcribing the speech of a first speaker, but poor resultswhen translating/transcribing the speech of a second speaker, even whenthe two speakers are speaking the same words. This may be caused, forexample, by different accents, intonation, speaking styles, etc., thatare specific to a particular speaker. Results may also vary based uponcontext. For example, the particular translation/transcription servicemay provide a high level of accuracy when translating/transcribinginformation in a first context, such as a medical context, but a lowlevel of accuracy when translating/transcribing information in a secondcontext, such as a computer science context. Consequently, no singletranslation/transcription service provides the best results in allcircumstances, i.e., across all languages, contexts, speakers, length ofphrases, etc.

Differences in accuracy of different translation/transcription servicesmay be attributable to a wide variety of factors, such as the design ofthe algorithms used, how an algorithm is trained, etc. For example, analgorithm for a particular translation/transcription service may beoptimized for a particular language and/or context, and not provideaccurate results for other languages and contexts. As another example,the training data set used with a particular translation/transcriptionservice may be focused on a particular language or context, whichreduces accuracy. Attempts are sometimes made to improve accuracy byre-running algorithms multiple times and/or retraining algorithms usingdifferent and/or more extensive data sets, both of which necessarilyconsume additional computational and storage resources, and requireadditional time, without necessarily improving results.

According to one embodiment, multiple translation/transcription servicesare utilized to improve translation/transcription results. Thetranslation/transcription services are selected, from a plurality ofavailable translation/transcription services, to improve accuracy incertain circumstances. The selection may be made based upon variousfactors that may include, without limitation, language, context, speakeridentification, class of speaker, location, compliance requirements, andother factors that may vary depending upon a particular implementation.Translation/transcription results provided by thetranslation/transcription services are selectively combined to generateresulting translation/transcription data that has a higher level ofaccuracy that can be achieved using conventional approaches that employa single translation/transcription service. Other benefits of theapproach are described hereinafter. Supplemental processing may also beperformed to further improve the accuracy of the resultingtranslation/transcription data.

A. Architecture

FIG. 10 is a block diagram that depicts an arrangement 1000 forperforming translation/transcription using multipletranslation/transcription services, according to an embodiment.Arrangement 1000 includes client device 1010, atranslation/transcription server 1020, and translation/transcriptionservices 1030 communicatively coupled via a network 1040. Network 1040may be implemented by one or more networks, such as one or more localarea networks (LANs), wide area networks (WANs), etc. The elements inarrangement 1000 may also have direct connections and embodiments arenot limited to the example elements depicted in arrangement 1100 andfewer or additional elements may be used, depending upon a particularimplementation.

Client device 1010 may be any type of device that is capable ofrequesting the translation and/or transcription of data. Examples ofclient device 1010 include, without limitation, desktop computers,notebook computers, tablet computing devices, mobile communicationsdevices such as smartphones, MFPs, nodes 104A-104N, meeting intelligenceapparatus 102, etc. Client device 1010 may include various processesused to support electronic meetings. For example, client device 1010 mayinclude a collaboration client that allows a user of client device 1010to participate in an electronic meeting. As another example, clientdevice 1010 may include a collaboration server process that manageselectronic meetings between participants.

Translation/transcription server 1020 acts as an intermediary betweenclient device 1010 and translation/transcription services 1030, andmanages the processing of translation/transcription requests bytranslation/transcription services 1030 on behalf of client device 1010.As described in more detail hereinafter, this includes causing a requestto translate/transcribe data to be processed by multipletranslation/transcription services 1030 and selectively combining theresults of the translation/transcription services 1030 into resultingtranslation/transcription data.

Translation/transcription server 1020 may be implemented by any type ofcomputing device and may include one or more software processes. Assuch, translation/transcription server 1120 is not limited to a serverper se, but may be implemented as any type of intermediate service. Inaddition, although depicted in the figures as a separate entity,translation/transcription server 1020 may be implemented on clientdevice 1010, or other entities, such as client devices, MFPs, nodes104A-104N, and meeting intelligence apparatus 102.

Translation/transcription services 1130 include services with thecapability to translate text data from one language to one or more otherlanguages, transcribe audio data into text, or both.Translation/transcription services 1130 may be implemented, for example,as Web applications or other processes on servers or other networkingelements and translation/transcription services 1130 may support one ormore application programming interfaces (APIs).

FIG. 11 is a block diagram that depicts an arrangement 1100 forperforming translation/transcription using multipletranslation/transcription services, according to an embodiment.Arrangement 1100 includes requestors 1110, 1120, atranslation/transcription server 1130, and translations/transcriptionservices 1168 that include translations/transcription services 1170,1180, 1190 through N. The elements depicted in FIG. 11 may becommunicatively coupled via any type and number of networks andcommunications links, including direct links. Embodiments are notlimited to the example elements depicted in arrangement 1100 and feweror additional elements may be used, depending upon a particularimplementation. For example, embodiments are applicable to any number ofrequestors and translation/transcription services.

Requestors 1110, 1120, also referred to herein as “requestor devices,”may be any entity that requests translation/transcription of data.Examples of requestors 1110, 1120 include, without limitation, clientdevices including desktop computers, notebook computers, tabletcomputing devices, mobile communications devices such as smartphones,MFPs, nodes 104A-104N, meeting intelligence apparatus 102, acollaboration server, a collaboration client, etc.

Translation/transcription server 1130 acts as an intermediary betweenrequestors 1110, 1120 and translation/transcription services 1170, 1180,1190 through N (collectively referred to in FIG. 11 astranslation/transcription services 1168) to providetranslation/transcription data on behalf of requestors 1110, 1120 usingmultiple translation/transcription services. This includestranslation/transcription server 1130 selecting particulartranslation/transcription services from availabletranslation/transcription services 1168 to provide the most accuratetranslation/transcription results for a particular situation.Translation/transcription server 1130 selectively uses thetranslation/transcription results provided by the selectedtranslation/transcription services to generate resultingtranslation/transcription data 1114, 1124, which is provided back torequestors 1110, 1120. As described in more detail hereinafter, theresulting translation/transcription data 1114, 1124 may includetranslation/transcription data from multiple translation/transcriptionservices.

According to one embodiment, translation/transcription server 1130includes a translation/transcription manager 1140 that implements one ormore heuristics, decision logic, algorithms, intelligence, naturallanguage processing, natural language classifying, learning, etc., toperform the functionality described herein. In addition, as described inmore detail hereinafter, the heuristics, decision logic, algorithmsetc., used by translation/transcription manager 1140 may be dynamic andchange over time via user feedback.

Translation/transcription server 1130 may be implemented by and on anytype of computing device and may include one or more software processes.As such, translation/transcription server 1130 is not limited to aserver per se, but may be implemented as any type of intermediateservice. In addition, although depicted in the figures as a separateentity, translation/transcription server 1130 may be implemented onrequestors 1110, 1120, or other entities, such as client devices, MFPs,nodes 104A-104N, and meeting intelligence apparatus 102.

Translation/transcription services 1168 include services with thecapability to translate text data from one language to one or more otherlanguages, transcribe audio data into text, or both.Translation/transcription services 1168 through N may be implemented,for example, as Web applications or other processes on servers or othernetworking elements and translation/transcription services 1168 maysupport one or more application programming interfaces (APIs).

B. Selection Data

According to one embodiment, translation/transcription manager 1140 usesselection data 1152, maintained as part of configuration data 1150, toselect particular translation/transcription services, from a pluralityof available translation/transcription services, to translate/transcribetext or audio data. The selected translation/transcription services areselected in a manner to provide the most accurate results for particulartext or audio data, and the translation/transcription services that areselected may be different for each set of text and audio data.

FIG. 12 depicts example data that may be included in selection data1152. In the examples depicted in FIG. 12, the labels “S1”, “S2”, and“S3” refer to three different translation/transcription services, suchas translation/transcription services 1170, 1180, 1190, but data may beprovided for any number of translation/transcription services. Each ofthe tables in FIG. 12 identifies one or more particulartranslation/transcription services that are likely to provide accurateresults for a particular factor, such as language, context, speaker,location, and for a particular compliance requirement. A context table1210 specifies one or more translation/transcription services that arelikely to provide accurate results for a particular context. Forexample, for text or audio data in the medical context,translation/transcription service S1 is most likely to provide accurateresults, while in the scientific context, translation/transcriptionservices S2 and S3 are most likely to provide accurate results.

A speaker table 1220 specifies one or more translation/transcriptionservices that are likely to provide accurate results for a particularspeaker. For example, translation/transcription services S1 and S2 arelikely to provide accurate results when processing text or audio data,or a portion thereof, that corresponds to “Speaker A” speaking, whiletranslation/transcription service S3 is likely to provide accurateresults when “Speaker D” is speaking.

A compliance table 1230 specifies one or more translation/transcriptionservices to be used when a particular compliance requirement applies. Acompliance requirement may correspond to a legal requirement that may beadopted, for example, by business organizations, countries, etc. Forexample, a particular country may adopt a legal requirement thatcorresponds to compliance requirement A, for which all oftranslation/transcription services S1, S2, and S3 are likely to provideequally accurate results. Another country, however, may adopt a legalrequirement that corresponds to compliance requirement D, for whichtranslation/transcription service S1 is likely to provide accurateresults. In this example, the legal requirement may prohibit the use oftranslation/transcription services that are outside the country and onlytranslation/transcription service S1 is located within the country.

A location table 1240 specifies one or more translation/transcriptionservices that are likely to provide accurate results for a particularlocation. As used here, the term “location” refers to a physicallocation that corresponds to text or audio data to be translated and/ortranscribed. For example, for text or audio data corresponding tolocation “A”, translation/transcription service S3 is likely to provideaccurate results, while for text or audio data captured in a location“B”, translation/transcription service Si is likely to provide accurateresults. This may be useful in situations where particulartranslation/transcription services are known to provide accurate resultsin certain locations attributable, for example, to local differences inlanguage.

A language table 1250 specifies one or more translation/transcriptionservices that are likely to provide accurate results for a particularcombination of source language and target language. For example, whentranscribing English text or audio data to English,translation/transcription service S2 is likely to provide accurateresults, when translating/transcribing from Japanese to English,translation/transcription service S1 is likely to provide accurateresults, and when translating/transcribing from German to French,translation/transcription services S2 and S3 are likely to provideaccurate results.

The particular contexts, speakers, compliance groups and languagesdepicted in FIG. 12 are provided for explanation purposes only andembodiments are not limited to the examples depicted in FIG. 12, andother factors may be used, depending upon a particular implementation.One example of another factor is classes of users, such as non-nativespeakers of a particular language, e.g., non-native English speakers.Other non-limiting examples of factors include history, reputation anddomain. Examples of domain include, without limitation, legal (patents,criminal, civil), medical, engineering (software, hardware, agileprocesses), business (marketing, sales, finance), and scientificresearch.

Selection data 1152 may be determined in different ways that may varydepending upon a particular implementation. For example, selection data1152 may be initially specified manually by a user, such as anadministrator, and then manual confirmation and/or statistical analysismay be used to establish that a particular translation/transcriptionservice provides accurate results for a particular factor, e.g., resultsthat have an accuracy above a specified threshold. For example, it maybe determined by manual confirmation and/or statistical analysis thattranslation/transcription service S1 provides accurate results whenbeing used to translate/transcribe text or audio data in the medicalcontext. As another example, it may be determined by manual confirmationand/or statistical analysis that translation/transcription services S1,S2 and S3 all provide accurate results when being used totranslate/transcribe text or audio data of Speaker A speaking, whileonly translation/transcription service S3 provides accurate results whenSpeaker D is speaking. Differences in results may occur for differentreasons. For example, a particular speaker may have an accent that ismore accurately translated/transcribed by one translation/transcriptionservice over another translation/transcription service. As anotherexample, one translation/transcription service may provide more accurateresults than another translation/transcription service when translatingtext or audio data in a particular context. Differences in accuracyprovided by different translation/transcription services for differentfactors (context, speaker, compliance, language, etc.) may beattributable to various reasons. For example, algorithms used bytranslation/transcription services may be optimized for particularfactors. As another example, even if the algorithms used by differenttranslation/transcription services are comparable, e.g., implementcomparable logic, there may be differences in the training set, i.e.,the amount and/or type of training data, used to train the algorithms.Selection data 1152 may be revised over time based upon user feedbackincluded in learning data 1158. For example, the preferred services inthe tables of FIG. 12 may be revised, e.g., “fine-tuned,” over time inresponse to actual experiences of a business organization in the form ofuser confirmations and/or corrections represented by learning data 1158.

Selection data 1152 may be used by translation/transcription manager1140 in different ways that may vary depending upon a particularimplementation. Translation/transcription server 1130 may use selectiondata 1152 to exclude the use of certain translation/transcriptionservices. For example, when translating/transcribing text or audio datafrom English to Japanese, translation/transcription manager 1140 mayonly use translation/transcription service S1 and excludetranslation/transcription services S2 and S3. Translation/transcriptionmanager 1140 may instead use selection data 1152 to identify preferencesamong translation/transcription services, without necessarily excludinga particular translation/transcription service that is not identified aspreferred for a particular factor and circumstance. For example, theresults of translation/transcription services that are identified aspreferable for a particular factor and circumstance may be given moreweight, i.e., more influence, than results of translation/transcriptionservices that are not identified as preferable for a particular factorand circumstance. Continuing with the prior example, whentranslating/transcribing text or audio data from English to Japanese,translation/transcription manager 1140 may apply a greater weighting toresults provided by translation/transcription service S1 and a lesserweighting to the results provided by translation/transcription servicesS2 and S3.

Translation/transcription manager 1140 may also use multiple factors inselection data 1152 to determine which results oftranslation/transcription services should be used. For example, text oraudio data that pertains to the sports context (S1, S2) and by speaker A(S1, S2) in location B (S1), the results of translation/transcriptionservice S1 may be used. Alternatively, the results of bothtranslation/transcription service S1 and S2 may be used, but the resultsof translation/transcription service S1 weighted more heavily than theresults of translation/transcription service S2. Also, in somesituations text or audio data may be translated/transcribed intomultiple target languages. This may occur, for example, duringelectronic meetings that are conducted in multiple languages. In thesesituations, multiple language factor values may be used bytranslation/transcription manager 1140 to determine whichtranslation/transcription services should be used.

C. Generating Translation/Transcription Data

FIG. 13 is a flow diagram 1300 that depicts an approach for generatingtranslation/transcription data using multiple translation/transcriptionservices, according to an embodiment. For purposes of explanation, it ispresumed that requestor 1110 makes the request 1112 fortranslation/transcription data to translation/transcription manager1140, and that translation/transcription manager 1140 generates andprovides the resulting translation/transcription data 1114 to requestor1110.

In step 1302, a request is received to translate/transcribe data. Forexample, translation/transcription manager 1140 receives, from requestor1110, a request to translate/transcribe data. The request specifies orincludes text or audio data to be translated/transcribed. For example,the request may include text or audio data to be translated/transcribed,or a reference, such as a link, to the text or audio data to betranslated/transcribed. The text or audio data may represent a completedconversation or session, such as a meeting, conversation, etc.Alternatively, the text or audio data may represent a portion of aconversation or session, and translation/transcription manager 1140 mayprocess the text or audio data as it is generated. For example,translation/transcription manager 1140 may process text or audio data asit is generated and streamed during an electronic meeting to providelive translation/transcription services during an electronic meeting.Embodiments are not limited to requestors issuing explicit, separaterequests for each portion of text or audio data to betranslated/transcribed, and providing text or audio data may itselfconstitute a request to translate/transcribe data. For example,requestor 1110 may send a first request that represents the start of astream of text or audio data, followed by multiple subsequent requeststhat represent chunks of text or audio data that comprise the stream.

Audio and text data may be in any format that may vary depending upon aparticular implementation. Example audio data formats include, withoutlimitation, MP3 (.mp3), Windows Media Audio (.wma), Real Audio (.ra,.ram, .rm), MIDI (.mid) and Ogg (.ogg) formats. The request may alsospecify the type of data desired, e.g., translated text or audio data,transcription data, or both translated audio data and transcriptiondata, as well as the format of the translated text or audio data, ortranscription data.

In step 1304, translation/transcription manager 1140 processes therequest and selects translation/transcription services to be used toprocess the text or audio data. The selection oftranslation/transcription services may be made in different ways andbased upon a variety of factors that may vary depending upon aparticular implementation. For example, translation/transcriptionservices may be selected randomly. Without any prior knowledge of thelevel of accuracy that any particular translation/transcription servicewill provide for a particular situation, this approach assumes thatgiven a large enough sample of translation/transcription services, theaccuracy of the resulting translation/transcription data will be betterthan when using only a single translation/transcription service,particularly when results from multiple translation/transcriptionservices are combined, as described in more detail hereinafter.

According to one embodiment, translation/transcription manager 1140 usesselection data 1152 to select particular translation/transcriptionservices, as previously described herein, to obtain a higher level ofaccuracy for a particular situation defined by factors, such as context,speaker, compliance, location, language, etc. The values of the factors,i.e., the particular language, particular speaker, etc., may be manuallyspecified by a user, for example, via a configuration interface providedby translation/transcription manager 1140, or via an interface providedby requestor 1110. Such a configuration interface may allow a user tospecify, for example via a graphical user interface with variouscontrols, a particular context, one or more speakers, a compliancegroup, a location and a source and target language for text or audiodata to be processed. Alternatively, the request received from requestor1110 may specify values for the factors, such as a language, context,speaker, etc. As another alternative, translation/transcription manager1140 may determine the values for the various factors. For example, whenthe text or audio data corresponds to an electronic meeting,translation/transcription manager 1140 may determine the value forvarious factors by examining data for the electronic meeting, such as anagenda, meeting notes/minutes, participant information, etc. As anotherexample, translation/transcription manager 1140 may examine the text oraudio data and/or metadata for the text or audio data. For example,translation/transcription manager 1140 may examine and/or analyze theaudio data to determine one or more of a context, one or more speakers,the applicability of a compliance group, a location, a source language,and one or more target languages. Any combination of the foregoing maybe used by translation/transcription manager 1140 to determine thevalues for the factors.

Once the values of the factors are known, translation/transcriptionmanager 1140 uses selection data 1152 to selecttranslation/transcription services that are most likely to provideaccurate results. For example, if the text or audio data is in Englishand is to be translated/transcribed to Japanese, then service S1 isselected. If the speaker is known to be speaker A, then service S1 isselected. If, however, the speaker is known to be speaker C, thenservices S1 and S2 are selected.

The number of translation/transcription services used may vary dependingupon a particular implementation, and embodiments are not limited to anyparticular number of translation/transcription services. Various factorsmay be considered in determining the number of translation/transcriptionservices to be used in a particular situation. Example factors include,without limitation, a desired level of accuracy, the availability ofcomputational resources, a contracted level of service, etc. Forexample, for applications that require a high level of accuracy andreal-time processing, a large number of translation/transcriptionservices may be used. For non-real-time processing, such as stand-alonetranslation and transcription situations, fewertranslation/transcription services may be used. The number oftranslation/transcription services used may be specified manually, orfor example, specified by selection data 1152.

The number of translation/transcription services used may tailored,i.e., specifically selected, for each set of text or audio dataprocessed. For example, a large number of translation/transcriptionservices may be selected to process a set of text or audio data toobtain a high level of accuracy, while a fewer number oftranslation/transcription services may be used to process another set oftext or audio data where a lower level of accuracy is acceptable, orwhere a few number of translation/transcription services is specified bya particular level of service. Using fewer translation/transcriptionservices may reduce the amount of computational resources and/or timerequired to perform the translation/transcription.

The number of translation/transcription services may be dynamicallychanged while a particular set of text or audio data is being processed.For example, the number of translation/transcription services may beincreased if a current level of accuracy, determined during theprocessing of text or audio data, is determined to be unsatisfactory,e.g., less than a specified threshold level of accuracy. Similarly, thenumber of translation/transcription services being used may be reducedif the level of accuracy exceeds the specified threshold level ofaccuracy, or in response to a change in subscription or service level.For example, a particular, a particular translation/transcriptionservice that is providing a low level of accuracy may be discontinued.In the present example, for purposes of explanation, it is presumed thattranslation/transcription manager 1140 selects translation/transcriptionservices 1170, 1180, 1190 to process the text or audio data, from the Nnumber of available translation/transcription services.

In step 1306, the text or audio data is provided to the selectedtranslation/transcription services for processing. In the presentexample, translation/transcription manager 1140 provides the text oraudio data, or a link to the text or audio data, totranslation/transcription services 1170, 1180, 1190 for processing. Textor audio data may be provided to the selected translation/transcriptionservices via one or more requests, which may be the requests thattranslation/transcription server 1130 received from requestors 1110,1120, or modified or newly-generated requests that contain differentinformation than the requests received from requestors 1110, 1120. Therequests may conform to APIs supported by translation/transcriptionservices 1170, 1180, 1190.

In step 1308, the selected translation/transcription services processthe text or audio data and generate and providetranslation/transcription data to translation/transcription manager1140. In the present example, translation/transcription services 1170,1180, 1190 process the text or audio data and generatetranslation/transcription data 1172, 1182, 1192, respectively, which mayinclude various types of data and in different formats. The type andformat of translation/transcription data 1172, 1182, 1192 may vary foreach translation/transcription service and depending upon a particularimplementation, and embodiments are not limited to any particular typeor form of translation/transcription data. For example,translation/transcription data 1172, 1182, 1192 may include text datathat represents text that has been translated from one language to oneor more other languages. As another example, translation/transcriptiondata 1172, 1182, 1192 may include text data that represents a result oftranscribing human speech to text. Translation/transcription data 1172,1182, 1192 may also include both text data and audio data.

In step 1310, translation/transcription data is received from thetranslation/transcription services. For example,translation/transcription manager 1140 receivestranslation/transcription data 1172, 1182, 1192 fromtranslation/transcription services 1170, 1180, 1190.Translation/transcription manager 1140 may storetranslation/transcription data 1172, 1182, 1192 locally astranslation/transcription data 1160, and later process as described inmore detail hereinafter. After processing, translation/transcriptiondata 1172, 1182, 1192 may be maintained on translation/transcriptionserver 1130 or deleted, depending upon a particular implementation. FIG.14 is a block diagram that depicts examples of translation/transcriptiondata (TTD) 1172, 1182, 1192 and resulting translation/transcription data1114. Translation/transcription data 1172, 1182, 1192 each include aplurality of units of speech, referred to as UOS₁ through UOS_(N). Asused herein, the term “unit of speech” refers to one or more spokenwords or utterances that represent a single word, a phrase, a sentence,a paragraph, etc. Thus, different units of speech may include differentnumbers of words. A unit of speech may be determined from text or audiodata based upon, for example, pauses, changes in intonation, and/or anycombination of factors. For example, natural pauses in human speech,e.g., at the end of a sentence, paragraph, etc., may define transitionsbetween units of speech. An audio or text file may contain any number ofunits of speech.

According to one embodiment, each unit of speech has a correspondingconfidence score, referred to in FIG. 14 as CS₁ through CS_(N). Theconfidence scores may be determined, for example, bytranslation/transcription services 1170, 1180, 1190. A confidence scoreconveys an estimated level of accuracy in a translated/transcribed unitof speech. Confidence scores may be determined by atranslation/transcription service based upon the results of priortraining. In the example depicted in FIG. 14, confidence scores arepresented as a percentage on a scale of 0 to 100 for purposes ofexplanation only, but embodiments are not limited to this example andany confidence scoring methodology may be used. Confidence scores may berepresented by confidence score data that may be maintained as part of,or separate from, units of speech, and confidence scores are depicted inFIG. 14 as being part of each unit of speech for purposes of explanationonly.

Translation/transcription data 1172, 1182, 1192 may also includemetadata that specifies one or more attributes oftranslation/transcription data 1172, 1182, 1192, such as timestampinformation, data size information, information about content, includingcontext, speaker, location, etc. The metadata may include data that ismanually specified by a user, for example during configuration ortraining, and data that is determined by translation/transcriptionservices 1170, 1180, 1190.

In step 1312, translation/transcription manager 1140 generates resultingtranslation/transcription data 1162 based upon thetranslation/transcription data 1172, 1182, 1192 received fromtranslation/transcription services 1170, 1180, 1190. This may beperformed in different ways that may vary depending upon a particularimplementation, and embodiments are not limited to any particularapproach for generating resulting translation/transcription data 1162.According to one embodiment, for each unit of speech,translation/transcription manager 1140 compares the confidence scoresfrom translation/transcription services 1170, 1180, 1190 and selects theversion of the translated/transcribed unit of speech with the highestconfidence score for the unit of speech. For example, referring to FIG.14, translation/transcription manager 1140, starting with the first unitof speech (UOS₁), compares the confidence score CS₁ fromtranslation/transcription services 1170, 1180, 1190 and determines thatthe translated/transcribed version of the first unit of speech (UOS₁)provided by translation/transcription service 1170 has the highestconfidence score of 92%, compared to the confidence scores of 82% and81%, respectively, for UOS₁ provided by translation/transcriptionservices 1180 and 1190. Translation/transcription manager 1140 thereforeincludes the translated/transcribed version of the UOS₁ provided bytranslation/transcription service 1170 in the resultingtranslation/transcription data 1162. This may be performed, for example,by translation/transcription manager 1140 invoking functionality tocause the translated/transcribed version of the first unit of speech(UOS₁) provided by translation/transcription service 1170 to be copiedand/or moved into resulting translation/transcription data 1162, whichmay be stored in a memory area designated for resultingtranslation/transcription data. This process is performed for each ofthe other units of speech until all of the units of speech have beenprocessed.

According to one embodiment, a confidence score threshold is used incombination with selection data 1152 to select translated/transcribedunits of speech to be included in resulting translation/transcriptiondata. In this embodiment, if all of the confidence scores for aparticular unit of speech are below a specified threshold, then insteadof using the translated/transcribed unit of speech having the highestconfidence score, selection data 1152 is used to select thetranslated/transcribed unit of speech that is likely to provide the mostaccurate result. For example, the various factors specified by selectiondata 1152 may be prioritized in the order of language, context, speaker,compliance requirement, and location. For this example, language table1250 is first consulted to determine whether a particulartranslation/transcription service is preferred over the othertranslation/transcription services that were used. If so, then thetranslated/transcribed unit of speech from the particulartranslation/transcription service is used. If language table 1250indicates that multiple translation/transcription services arepreferred, then the next highest priority factor is considered. In thepresent example, the context table 1210 is next consulted to determinewhich of the translation/transcription services identified by thelanguage table 1250 as preferred should be used. For example, supposethat both the translation/transcription services S1 and S3 are specifiedas preferred by language table 1250. If the context table 1210 specifiesthat translation/transcription service S1 is preferred, buttranslation/transcription service S3 is not, then thetranslated/transcribed unit of speech from translation/transcriptionservice S1 is used. This process may be repeated with additional factorswith lower priority until a result from a particulartranslation/transcription service is selected. Alternatively, units ofspeech for which all of the confidence scores are below a specifiedthreshold may be designated for supplemental processing, as described inmore detail hereinafter.

As depicted in FIG. 14, resulting translation/transcription data 1162includes translation/transcription data from all threetranslation/transcription services that is interleaved, although this isnot required and may not occur in every situation. For example, in somesituations a particular translation/transcription service may provideunexpectedly low accuracy across all units of speech for particular textor audio data, and in these situations, none of the results provided bythe particular translation/transcription service may be used, eventhough, based upon the selection data, the particulartranslation/transcription service was expected to provide accurateresults. In other situations, resulting translation/transcription data1162 may include a large number of contiguous translated/transcribedunits of speech from a first translation/transcription service, followedby a large number of contiguous translated/transcribed units of speechfrom a second translation/transcription service. This may occur, forexample, if the first translation/transcription service providedaccurate results for a particular context, speaker, language, etc., butthe accuracy level decreased after a change in context, speaker,language, etc. At that point, a second translation/transcription serviceprovided more accurate results, so the translated/transcribed units ofspeech from the second translation/transcription service were selectedfor inclusion in the resulting translation/transcription data. Thus,translation/transcription manager 1140 may at any time switch betweentranslation/transcription services based upon the accuracy of thetranslated/transcribed results. In situations, resultingtranslation/transcription data 1162 may include translated/transcribedunits of speech from only a single translation/transcription service,where the confidence scores for that translation/transcription servicewere the highest across all units of speech.

As depicted in FIG. 14, the confidence scores may for eachtranslated/transcribed unit of speech may be included in resultingtranslation/transcription data 1162 and may be used during subsequentprocessing. For example, confidence scores may be displayed on agraphical user interface concurrent with the display of resultingtranslation/transcription data 1162 to provide a visual indication ofthe accuracy of translated/transcribed units of speech. Confidencescores may also be included in various types of documentation, such as ameeting summary or meeting transcript. Special visual effects, such ashighlighting, formatting, etc., may be used to conspicuously identifyparticular units of speech that have a confidence score below aspecified threshold, to trigger, for example, manual editing or specialprocessing. For example, suppose that the confidence scores for aparticular unit of speech from all of the translation/transcriptionservices that were used are below a specified threshold. This may causetranslation/transcription manager 1140 to designate the particular unitof speech for manual editing. According to one embodiment,translation/transcription manager 1140 provides a graphical userinterface that allows manual editing of translation/transcriptionresults and that includes visually identifying units of speech that havebeen designated for manual editing based upon confidence scores toimprove accuracy. The results of the manual editing may be stored inlearning data 1158 and included as an additional input totranslation/transcription manager 1140 to improve the accuracy of futuretranslations/transcriptions. Confidence scores may also be used by otherapplications to improve accuracy.

As depicted in FIG. 14, resulting translation/transcription data 1162may have corresponding metadata that specifies or more attributes ofresulting translation/transcription data 1162 such as the source of eachof the units of speech, i.e., translation/transcription data 1172, 1182,or 1192, data size information, information about content, includingcontext, speaker, location, etc. The metadata may be generated bytranslation/transcription services 1170, 1180, 1190, and/or bytranslation/transcription manager 1140, and stored as part of orseparate from translation/transcription data 1172, 1182, or 1192 andresulting translation/transcription data 1162. Resultingtranslation/transcription data 1162 may be stored locally attranslation/transcription server 1130 and/or stored in a location remoteto translation/transcription server 1130. Once resultingtranslation/transcription data 1162 has been generated, thetranslation/transcription data used to generate resultingtranslation/transcription data 1162 may be deleted fromtranslation/transcription data 1160 or retained for later use, dependingupon a particular implementation.

The generation of resulting translation/transcription data may beperformed in real-time as translation/transcription data is receivedfrom translation/transcription services or alternatively, after all ofthe processing has been performed by the translation/transcriptionservices and all of the translation/transcription data has beenreceived. For example, translation/transcription manager 1140 maygenerate resulting translation/transcription data 1162 in real-time astranslation/transcription data 1172, 1182, 1192 is received fromtranslation/transcription services 1170, 1180, 1190. Alternatively,translation/transcription manager 1140 may generate resultingtranslation/transcription data 1162 after all oftranslation/transcription data 1172, 1182, 1192 has been generated andreceived from translation/transcription services 1170, 1180, 1190.

D. Supplemental Processing

According to one embodiment, translation/transcription manager 1140 isconfigured to provide optional supplemental processing to improveaccuracy. Continuing with the flow diagram 1300 of FIG. 13, in step1314, translation/transcription manager 1140 performs supplementalprocessing on resulting translation/transcription data 1162 before it issupplied to requestor 1110. Supplemental processing may be performed onparticular units of speech, or groups of units of speech (including allunits of speech), to improve accuracy. Units of speech may be selectedfor supplemental processing based upon confidence scores. For example,according to one embodiment, units of speech having a confidence scorebelow a specified threshold are selected for supplemental processing.

Supplemental processing may include different types of processing thatmay vary depending upon a particular implementation. For example,auto-correction may be used to process units of speech designated forsupplemental processing. Auto-correction may be performed by one or morealgorithms that are designed to make corrections based upon statisticalmodeling. Statistical modeling may be used to identify errors in unitsof speech, such as grammatical errors, improper structure, impropersyntax, multiple nouns in a row, etc. The type of statistical modelingand the parameters for statistical modeling may be specified by thelogic, heuristics, algorithms, etc., used to implement auto-correction,and they may change over time based upon user feedback, such as usercorrections. The identified units of speech are then corrected basedupon the results of the statistical modeling to improve accuracy.

According to one embodiment, auto-correction includes using knowncorrect units of speech to correct and/or supplement units of speechthat have been designated for supplemental processing. This may includedetermining particular known correct units of speech that are similar tothe units of speech that have been designated for supplementalprocessing, and then updating the units of speech that have beendesignated for supplemental processing based upon the particular knowncorrect units of speech. The particular known correct units of speechmay be specified, for example, by one or more databases, dictionaries,etc.

The determination of similarity may be made using a wide variety oftechniques that may vary depending upon a particular implementation.According to one embodiment, the determination of similarity is madeusing distance. Any type of heuristic and/or algorithm may be used todetermine the distance between units of speech designated forsupplemental processing and particular known correct units of speech,and embodiments are not limited to any particular heuristic and/oralgorithm. Non-limiting examples include a Levenshtein distance, aHamming distance (for units of speech of equal length) and aJaro-Winkler distance. The distance or similarity thresholds used totrigger auto-correction may vary depending upon a particularimplementation and may be specified by configuration data 150. Distanceor similarity thresholds may be associated with different levels ofauto-correction that are manually selectable and/or specified byconfiguration data 150. For example, a high level of auto-correction,i.e., more aggressive, may have a corresponding large distance, while alow level of auto-correction, i.e., less aggressive, may have acorresponding small distance. In addition, the statistical analysis usedmay consider other information to improve accuracy, such as surroundingunits of speech, etc.

According to one embodiment, auto-correction includes making changes tounits of speech based upon organization/industry-specific informationthat specifies commonly-used terms for an organization or industry. Asused herein, the term “organization” refers to any organized group ofpeople with a particular purpose. Examples of organizations include,without limitation, business entities, societies, schools, socialgroups, etc. Commonly-used terms for an organization may include, forexample, the name of the organization, names of products and/or servicesprovided by the organization, names of divisions, departments, projects,people, etc., within the organization. As used herein, the term“industry” refers to a particular form or branch of economic orcommercial activity. Organization/industry-specific information mayinclude “coined” terms that have a special meaning within a particularorganization/industry, but not necessarily to otherorganizations/industries. Organization/industry-specific data 1154 maybe maintained as part of configuration data 1150, or may be maintainedseparate from selection configuration 1150.Organization/industry-specific data 1154 may be initially manuallycreated by a user, such as an administrative user, and then revised overtime to further improve accuracy. Organization/industry-specific data1154 may include standard terms that may be customized by particularorganizations and/or industries.

According to one embodiment, statistical analysis is used to identifyportions of units of speech, and/or entire units of speech, that arelikely to be a commonly-used term for an organization or industry. Thismay include using one or more algorithms for calculating the distance,i.e., similarity, between a particular unit of speech, or a portionthereof, and one or more terms from organization/industry-specific data1154. For example, suppose that a particular translated/transcribed unitof speech is “multi-function parade” and one of theorganization-specific terms included in organization/industry-specificdata 1154 is “multi-function peripheral.” In this example, the term“peripheral” is misconstrued as “parade.” Statistical analysis maydetermine that the distance between the units of speech “multi-functionparade” and “multi-function peripheral” is so small, i.e., that theterms are so similar, that there is a high likelihood that the originalspoken or written unit of speech was actually intended to be“multi-function peripheral,” but because of limitations in thetranslation/transcription process, resulted in thetranslated/transcribed unit of speech “multi-function parade.” In thissituation, auto-correction replaces or corrects thetranslated/transcribed unit of speech “multi-function parade” with“multi-function peripheral” to improve accuracy.

As another example, a particular business organization may sell aproduct named the “Quick Draw Analyzer” that is incorrectlytranslated/transcribed as “Quick Drawn Analyzer,” i.e., with an extra“n” at the end of the term “Draw.” Statistical analysis may determinethat the distance between the unit of speech “Quick Drawn Analyzer” andthe name of the business organization's “Quick Draw Analyzer” product isso small, i.e., that the units of speech are so similar, that is highlylikely that the translated/transcribed unit of speech “Quick DrawnAnalyzer” was intended to be “Quick Draw Analyzer.” In this situation,the translated/transcribed unit of speech “Quick Drawn Analyzer” ismodified to “Quick Draw Analyzer” to improve accuracy.

As yet another example, the unit of speech “Ricoh,” referring to thewell-known international corporation, may be mistakenlytranslated/transcribed to “Rico” because of limitations intranslation/transcription algorithms and/or training data. In thisexample, the letter “h” is left off the end of “Rico.” Supplementalprocessing may determine that there is a very high likelihood, basedupon organization-specific information, that the unit of speech “Rico”should instead be “Ricoh,” and so during supplemental processing, theunit of speech “Rico” is changed to “Ricoh.”

The auto-correction approaches described herein may be applied toportions of units of speech, or entire units of speech. This may includedecomposing a unit of speech into its constituent components and thenperforming auto-correction on the constituent components, either aloneor in groups. This may be particularly beneficial for large units ofspeech, i.e., units of speech that include a large number of words.Auto-correction may be performed on different combinations ofconstituent components of a unit of speech. For units of speech thatinclude many words, decomposition may be performed iteratively untilfurther decomposition is not possible. In addition, although embodimentsare described herein in the context of performing supplementalprocessing on units of speech, or constituent components thereof,embodiments are not limited to these examples and supplementalprocessing may be applied across multiple units of speech, e.g., ongroups of two or more adjacent units of speech. This may improveaccuracy for units of speech that contain on a few words.

The metadata for resulting translation/transcription data 1162 may beupdated to specify any changes made via supplemental processing. Forexample, the metadata may specify one or more units of speech that weremodified or substituted, the prior values of the units of speech,timestamp information that specifies when the changes were made, etc.

Continuing with the flow diagram 1300 of FIG. 13, after optionalsupplemental processing has been performed in step 1314, in step 1316,translation/transcription manager 1140 provides resultingtranslation/transcription data 1162 to requestor 1110 as resultingtranslation/transcription data 1114. Requestor 1110 may present theresulting translation/transcription data 1114 in real-time, for example,during an electronic meeting. This may include playing translated speechthrough one or more speakers and/or displaying transcribed text on oneor more displays. Alternatively, the translated and transcribed data maybe included in post meeting documents, such as meeting notes andreports, and also saved for later processing. Although embodiments aredescribed herein in the context of organization-specific information,embodiments are not limited to these examples and other types ofinformation may be used to improve the accuracy of resultingtranslation/transcription data.

Using the approach described herein and depicted in the figures,translation/transcription manager 1140 selects multipletranslation/transcription services that are most likely to provideaccurate results for a particular situation, i.e., a particular set offactors. Translation/transcription manager 1140 then selectively usesthe results provided by those translation/transcription services, on aunit of speech-by-unit of speech basis. The translation/transcriptionmanager 1140 may also perform supplemental processing to further improvethe accuracy of the resulting translation/transcription data. Thisprovides translated/transcribed information with a higher level ofaccuracy that can be achieved using conventional approaches that employa single translation/transcription service. In addition, the approachreduces the amount processing resources, storage resources, networkingresources, network traffic, and/or time attributable to performingadditional training of and re-running translation/transcriptionservices. For example, generating more accurate resultingtranslation/transcription data using multiple translation/transcriptionservices in a single iteration avoids expending computational andstorage resources on additional iterations, and also reduces networktraffic.

E. Multi-Language Electronic Meetings

The approaches described herein may be used to support electronicmeetings in which the participants have at least two different nativelanguages. Suppose that an electronic meeting includes two participantsand the first meeting participant's native language is language A andthe second participant's native language is language B. Suppose furtherthat the electronic meeting is to be conducted in language A. In thisexample, while the second participant may be able to generally speak andunderstand language A, the second participant may have difficultyspeaking particular words in language A, since language A is not thesecond participant's native language. In situations where theseparticular words are commands that are recognized by a computerizedsystem, such as a content collaboration system, it may be difficult forthe second participant to verbally command the system in language A byspeaking the particular words. Such systems typically respond to spokencommands recognized from recorded audio data or meeting transcripts, andrely upon the commands being spoken within a specified tolerance of anexpected pronunciation. The result is that mispronounced commands maynot be recognized and acted upon by these systems.

According to one embodiment, these issues are addressed by an approachthat generates command transcripts in multiple languages to allowparticipants to speak commands in their native language duringelectronic meetings. FIG. 15 is a block diagram that depicts an examplearrangement 1500 on which an embodiment may be implemented. Arrangement1500 includes two locations 1510, 1520 that correspond to participantsof a meeting. Locations 1510, 1520 may represent physical locations, forexample different offices within a business organization, with computerhardware and software that allows participants to participate in ameeting, such as an electronic collaboration meeting. Locations 1510,1520 may each include hardware and/or software for recording audio ofmeetings to enable audio data for meetings may be generated at one orboth of locations 1510, 1520. Example implementations of locations 1510,1520 include, without limitation, client devices, nodes 104A-104N, etc.Arrangement 1500 may include other elements, such as collaborationservers, etc., which are not depicted in FIG. 15 for purposes ofexplanation.

Each location has a corresponding default language that is the nativelanguage of one or more participants at the location. The defaultlanguage may be specified at each location by the participants, by ameeting organizer or an administrator, and default language is madeknown to translation/transcription server 1020/1130. For example, thedefault language for each location may be made known totranslation/transcription server 1020/1130 by client device 1010,requestors 1110, 1120, or an administrator, and may be specified byconfiguration data 1150. In the example depicted in arrangement 1500,the meeting is conducted in English, the default language for location1510 is English, and the default language for location 1520 is Japanese.

As depicted in FIG. 15, a meeting transcript is generated in English andis used, for example, to document the meeting, generate meetingsummaries, etc. According to an embodiment, a command transcript isgenerated in each of the default languages for the meeting, which in thepresent example are English and Japanese. Thus, in the present example,when translation/transcription server 1020/1130 receives and processes arequest to translate/transcribe data, translation/transcription server1020/1130 generates a meeting transcript in English and commandtranscripts in both English and Japanese. For example,translation/transcription data 1160 and/or resultingtranslation/transcription data 1162 may include data that represents ameeting transcript in English and command transcripts in both Englishand Japanese. The transcription data is made available to a commandprocessing system, for example, in a content collaboration system, forprocessing, so that commands may be recognized and acted upon inreal-time during a meeting.

As used herein, the term “meeting transcript” refers to a texttranscript of audio data for a meeting. A meeting transcript typicallyincludes everything that was said during a meeting, to the extent thatit can be determined from audio data for the meeting. A meetingtranscript may also include metadata for the meeting that may varydepending upon a particular implementation. Examples of metadatainclude, without limitation, a name, location, start time, duration,attendees, agenda items or topics, and action items for the meeting. Asused herein, the term “command transcript” refers to a text transcriptof commands spoken during a meeting. A command transcript may includeonly commands spoken during a meeting, and may include otherinformation, such as one or more portions (or all) of the conversationsthat occurred during the meeting. A command transcript may also includemetadata for the meeting as previously described herein with respect tothe meeting transcript. Command transcripts may be generated in a widevariety of ways that may vary depending upon a particularimplementation, and embodiments are not limited to any particularapproach for generating command transcripts. For example, a commandtranscript may be generated directly from audio data in the same manneras a meeting transcript. As another example, a command transcript may begenerated from a meeting transcript by removing non-command information.Specified commands may be included in configuration data 1150 and textthat is not one of the specified commands is removed from the meetingtranscript data to generate a command transcript. According to oneembodiment, the specified commands are recognizable by a contentcollaboration system. For example, the specified commands may includecommands for planning/scheduling or starting a meeting, adding/changingagenda items, adding or removing participants, specifying action items,specifying content to be shared in a meeting, etc. One of the benefitsof using command transcripts is that they may contain significantly lessdata than meeting transcripts and consequently, fewer computationalresources and less time is required to process and store commandtranscripts compared to meeting transcripts. In addition, the userexperience of manually reviewing a command transcript is greatlyimproved compared to looking for commands in a meeting transcript.

Meeting transcripts and command transcripts may be generated using atranslation/transcription service that is selected for the particularcombination of source and target languages. In the prior example inwhich the meeting is conducted in English, the default language forlocation 1510 is English, and the default language for location 1520 isJapanese, a first translation/transcription service may be used togenerate the meeting transcript and the command transcript for location1510 in English. Similarly, a second translation/transcription servicemay be used to generate the command transcript for location 1510 inJapanese. The first and second translation/transcription services may bethe same or different, and are selected to provide the most accurateresults. For example, referring to the language table 1250 of FIG. 12,translation/transcription service S2 may be used to generate the meetingtranscript and the command transcript for location 1510 in English,while translation/transcription service 51 may be used to generate thecommand transcript for location 1520 in Japanese, since these servicesare known and/or expected to provide accurate results for the particularcombination of source and target languages.

According to one embodiment, the translation/transcription services usedto generate the meeting transcript and the command transcripts in thedefault languages may be dynamically changed over time. For example, atranslation/transcription service currently being used may be changed ifthe confidence scores for the results no longer satisfy one or morespecified thresholds. In this situation, an alternativetranslation/transcription service is selected that is expected toprovide more accurate results. The change of a translation/transcriptionservice may be made at any time, including during a meeting and/or aftera meeting.

According to one embodiment, multiple translation/transcription servicesmay be used to generate the meeting transcript and the commandtranscripts in the default languages. According to this embodiment,audio data is processed by multiple translation/transcription servicesand the translation/transcription service having the high level ofaccuracy, as determined by confidence scores, is used to generatecommand transcripts. The translation/transcription service may bedynamically changed over time as confidence scores change. For example,if the confidence scores for a translation/transcription servicecurrently being used fall below a specified threshold, thentranscription may be switched to a different translation/transcriptionservice. Alternatively, multiple translation/transcription services maybe used and the results combined using confidence scores, as previouslydescribed herein. Using multiple translation/transcription services mayprovide results with a higher level of accuracy than using a singletranslation/transcription service. Also, identifying units of speech formanual processing, supplemental processing and auto-correctiontechniques previously described herein are also applicable to generatingcommand transcripts. According to one embodiment, multipletranslation/transcription services are used and if the confidence scoresfor all of the multiple translation/transcription services do notsatisfy a specified threshold, the results from multipletranslation/transcription services that were used are included in thecommand transcript, the meeting transcript, or both the commandtranscript and the meeting transcript. This may be performed on a unitof speech-by-unit of speech basis. For example, suppose that threetranslation/transcription services are being used to process audio dataand generate command transcripts. Suppose further that for a particularunit of speech, the confidence scores for all threetranslation/transcription services for the particular unit of speech areless than a specified threshold. According to one embodiment, more thanone of the results from the three translation/transcription services,i.e., multiple versions of the results, are included in the commandtranscript, the meeting transcript, or both the command transcript andthe meeting transcript. In the present example, this may include theresults from two or three of the three translation/transcriptionservices. Alternatively, the results from the top N highest scoringtranslation/transcription services are included in the commandtranscript. For example, suppose that the specified threshold is 76% andthe confidence scores from the three translation/transcription servicesfor the particular unit of speech are 71%, 34% and 58%, respectively. Inthis situation, the transcribed versions of the particular unit ofspeech from the first and third translation/transcription services areincluded in the command transcript, the meeting transcript, or both thecommand transcript and the meeting transcript. This may increase thelikelihood that a command is recognized by a command processing system.Additional information may be included in the command transcript toconspicuously identify the particular unit of speech. For example, oneor more special characters and/or special effects may be used toconspicuously identify the multiple versions of the particular unit ofspeech so that they may be readily identified by manual visualinspection.

Although embodiments are depicted in the figures and described herein inthe context of two locations and two default languages, one of which isthe same language as the language in which the meeting is conducted,embodiments are not limited to this example and are applicable to anynumber of locations and default languages. As one non-limiting example,the meeting transcript may be in a first language, the default languagefor location 1510 is a second language, and the default language forlocation 1520 is a third language. As another non-limiting example,suppose that arrangement 1500 included a third location with a defaultlanguage of French. In this situation, translation/transcription server1020/1130 generates a meeting transcript in English, and commandtranscripts in English, Japanese, and French.

According to an embodiment, real-time cross language support providesaudio and/or text to participants in their default language. Forexample, referring to FIG. 15, audio data in English from location 1510is translated and/or transcribed into Japanese when provided to location1520. Similarly, audio data in Japanese from location 1520 is translatedand/or transcribed into English when provided to location 1510. This maybe done using particular translation/transcription services or multipletranslation/transcription services, as previously described herein. Thetranslated audio may be played, for example, via one or more speakers,and the resulting text from transcribed audio may be displayed on a userinterface. This allows the participants to communicate in real-timeusing different languages and with more accuratetranslations/transcriptions using multiple translation/transcriptionservices as described herein.

FIG. 16 is a flow diagram that depicts an approach for providingmulti-language support for meetings in which the participants have atleast two different native languages. For purposes of explanation andwith reference to FIG. 11, it is presumed that requestor 1110 makes therequest 1112 for transcription data to translation/transcription manager1140, and that translation/transcription manager 1140 generates andprovides the resulting translation/transcription data 1114 to requestor1110. Requestor 1110 may be, for example, a collaboration clientexecuting on a client device, or a collaboration server, that needs tohave transcription data generated from meeting audio data.

In step 1602, a request is received to transcribe data. For example,translation/transcription manager 1140 receives, from requestor 1110, arequest to transcribe data. The request identifies and/or includes audiodata to be transcribed. The audio data may represent a portion of ameeting or an entire meeting. The request may also specify one or morelanguages for the transcribed data, and may also optionally specify thetype of transcription data desired, e.g., a meeting transcription,command transcriptions, or both a meeting transcription and commandtranscriptions. For example, requestor 1110 may send a first requestthat represents the start of a stream of text or audio data, followed bymultiple subsequent requests that represent chunks of text or audio datathat comprise the stream.

In step 1604, translation/transcription manager 1140 processes therequest and selects translation/transcription services to be used toprocess the text or audio data. As previously described, for each targetlanguage a single translation/transcription service may be selected thatis expected to provide the most accurate results. Alternatively,multiple translation/transcription services may be selected and theresults evaluated and selectively combined to provide the most accurateresulting transcription data, as previously described herein.

In step 1606, audio data is provided to the selectedtranslation/transcription services for processing. In the presentexample, for purposes of explanation, it is presumed thattranslation/transcription manager 1140 provides the audio data, or alink to the audio data, to translation/transcription service 1170 forprocessing.

In step 1608, the selected translation/transcription service processesthe audio data and generates and provides transcription data totranslation/transcription manager 1140. In the present example,translation/transcription service 1170 processes audio data andgenerates translation/transcription data 1172.

In step 1610, translation/transcription data is received from thetranslation/transcription services. For example,translation/transcription manager 1140 receivestranslation/transcription data 1172 from translation/transcriptionservice 1170. Translation/transcription manager 1140 may storetranslation/transcription data 1172 locally as translation/transcriptiondata 1160, and later process as described in more detail hereinafter.After processing, translation/transcription data 1172 may be maintainedon translation/transcription server 1130 or deleted, depending upon aparticular implementation.

In step 1612, translation/transcription manager 1140 generates resultingtranslation/transcription data 1162 based upon thetranslation/transcription data 1172 received fromtranslation/transcription service 1170. This may involve minimalprocessing if only a single translation/transcription service is used,as in the present example. If multiple translation/transcriptionservices are used, then generating resulting translation/transcriptiondata 1162 may be performed using confidence scores as previouslydescribed herein. Generating resulting translation/transcription data1162 may include generating a meeting transcript, and also one or morecommand transcripts that have non-command data removed.

In addition, since the resulting translation/transcription data 1162does not include translation data per se, in this example, the resultingtranslation/transcription data 1162 may be considered to betranscription data.

In step 1614, translation/transcription manager 1140 optionally performssupplemental processing on resulting translation/transcription data 1162before it is supplied to requestor 1110. Supplemental processing may beperformed on any portion of, or all of, resultingtranslation/transcription data 1162 to improve accuracy, as previouslydescribed herein.

In step 1616, translation/transcription manager 1140 provides resultingtranslation/transcription data 1162 to requestor 1110 as resultingtranslation/transcription data 1114. Requestor 1110 may present and/orprocess the resulting translation/transcription data 1114 in real-time,for example, during an electronic meeting. This may include, forexample, displaying the meeting transcript and/or the commandtranscripts. This may also include processing and taking actions basedupon commands contained in the command scripts. For example, the commandtranscripts may be provided to a collaboration management process thatrequested the transcripts, and the collaboration management process inturn provides the command transcripts to the meeting equipment atlocations 1510, 1520 so that appropriate action may be taken in responseto the commands contained in the command transcripts.

IX. Person Detection, Person Identification and Meeting Start

Various approaches are provided for performing person detection andidentification in conjunction with electronic meetings, both duringmeeting startup and ongoing meetings. As used herein, the termInteractive Whiteboard (IWB) appliance, also referred to as “interactivewhiteboards,” refers to electronic whiteboards that allow users to viewand update content on a display.

A. Architecture

FIG. 17A is a block diagram that depicts an arrangement 1700 formanaging electronic meetings according to one embodiment. Arrangement1700 includes IWB appliances 1710, 1720 and a meeting manager 1730,communicatively coupled via a network 1731. Network 1731 may be anycombination of wired or wireless networks including, for example, LANs,WANs, packet-switched networks such as the Internet, etc. IWB appliances1710, 1720 may be any type of IWB (similar or different) and the type ofIWB may vary depending upon a particular implementation. Two IWBappliances are depicted and described herein for purposes of explanationonly, and embodiments are applicable to any number of IWB appliances.Arrangement 1700 is not limited to the elements depicted in FIG. 17A,and may have fewer or more elements depending upon a particularimplementation.

IWB appliances 1710, 1720 may be implemented by any type of IWBappliance and may be configured with a wide variety of computerhardware, computer software, and other features that may vary dependingupon a particular implementation and embodiments are not limited to IWBappliances 1710, 1720 have any particular configuration and features.FIG. 17B is a block diagram that depicts an example implementation ofIWB appliance 1710, which may also be applicable to IWB appliance 1720.IWB appliance 1710 supports content collaboration sessions, such asvideoconferencing sessions, whiteboard/desktop sharing sessions, etc.,with other IWB appliances and other devices that support contentcollaboration sessions. Content collaboration sessions allow users toview, edit, and create content at either a single location or multiplelocations.

In the example depicted in FIG. 17B, IWB appliance 1710 includes adisplay 1740, microphones 1742, speakers 1744, cameras 1746, acommunications interface 1748, an IWB application 1750, a meetingmanager 1756, an operating system (OS) 1758, and a computingarchitecture 1760. IWB appliance 1710 may include fewer or morecomponents that those depicted in FIG. 1 and described herein, theparticular components used may vary depending upon a particularimplementation, and IWB appliance 1710 is not limited to a particularset of components or features. For example, IWB appliance 1710 may alsoinclude manual controls, such as buttons, slides, etc., not depicted inFIG. 1, for performing various functions on mobile device, such aspowering on/off or changing the state of IWB applicant 1710 and/ordisplay 1740.

Display 1740 may be implemented by any type of display that displayscontent, e.g., images, video, word processing documents, spreadsheets,etc., and other information to a user. Display 1740 may be capable ofreceiving user input and embodiments are not limited to any particularimplementation of display 1740. As one non-limiting example, display1740 may be a touchscreen display 1740 that both displays information tousers and allows user to select graphical user interface objects. IWBappliance 1710 may have any number of displays 1740, of similar orvarying types, located anywhere on IWB appliance 1710.

Microphones 1742 provide the capability for IWB appliance 1710 to detectand receive audio information from, for example, meeting participantsand other persons. The audio information may represent, for example,sounds, music, spoken words, etc. Speakers 1744 provide the capabilityfor IWB appliance 1710 to play audio information, such as sounds, music,speech, etc. Cameras 1746 provide the capability for IWB appliance 1710to detect and acquire image and/or video information. Microphones 1742,speakers 1744, and cameras 1746 may be implemented by any number andtype of microphones, speakers, and cameras, respectively, andembodiments are not limited to any particular type and number ofmicrophones, speakers and cameras.

Communications interface 1748 may include computer hardware, software,or any combination of computer hardware and software to support wiredand/or wireless communications between IWB appliance 1710 and otherdevices and/or networks, such as IWB appliance 1720 and meeting manager1730, as well as client devices (not depicted). The particularcomponents for communications interface 1748 may vary depending upon aparticular implementation and embodiments are not limited to anyparticular implementation of communications interface 1748.

IWB appliance 1710 also includes one or more applications that executeon IWB appliance 1710 and may each be implemented by one or moreprocesses. IWB application 1750 manages interactive content sessions,such as videoconferencing sessions, whiteboard/desktop sharing sessions,etc., with other devices, such as IWB appliance 1720. This may includethe capability to interact with one or more collaboration servers. Imagerecognition application 1752 determines the similarity of images, forexample, by comparing image data for a first image to image data for asecond image. Voice recognition application 1754 determines thesimilarity of voice data, for example, by comparing voice data for afirst set of one or more units of speech to a second set of one or moreunits of speech. IWB appliance 1710 may include additional applicationsand/or processes, that may vary depending upon a particularimplementation.

Meeting manager 1756 may be implemented by one or more processes thatmanage electronic collaboration meetings both on IWB appliance 1710 andwith other devices, such as IWB appliance 1720 and other devices. Thismay include initiating, managing and terminating electroniccollaboration meetings.

Operating system (OS) 1758 executes on computing architecture 1760 andmay be any type of operating system that may vary depending upon aparticular implementation and embodiments are not limited to anyparticular implementation of OS 1758. OS 1758 may include multipleoperating systems of varying types, depending upon a particularimplementation, and may include one or more APIs that provide access tofunctionality offered by OS 1758.

Computing architecture 1760 may include various elements that may varydepending upon a particular implementation and IWB appliance 1710 is notlimited to any particular computing architecture 1760. In the exampledepicted in FIG. 17B, computing architecture 1760 includes a processor1762 and storage 1764. Processor 1762 may be implemented by any numberand types of processors and storage 1764 may be implemented by anynumber and types of memories, including volatile memory and non-volatilememory, which may vary depending upon a particular implementation.Computing architecture 1764 may include additional hardware, firmwareand software elements that may vary depending upon a particularimplementation. According to one embodiment and as described in moredetail hereinafter, storage 1764 includes local copies of meetinginformation 1732 that specifies information for one or more meetings andidentity information 1734 that specifies information for one or moreusers. Storage 1764 may store other data of varying types, dependingupon a particular implementation.

Meeting manager 1730 is an entity that manages meeting information formeetings conducted between IWB appliances 1710, 1720. Meeting manager1730 may be implemented by computer hardware, computer software, or anycombination of computer hardware and computer software. For example,meeting manager 1730 may be implemented by one or more processesexecuting on one or more computing elements. Meeting manager 1730 isdepicted in FIG. 17A as a separate element for purposes of explanationonly, and the functionality of meeting manager 1730 may be implementedon other elements, including IWB appliances 1710, 1720, client devices(not depicted), etc. Meeting manager 1730 manages information pertainingto electronic meetings on behalf of IWB appliances 1710, 1720. Accordingto one embodiment, meeting manager 1730 manages meeting information 1732that specifies information for one or more meetings and identityinformation 1734 that specifies information for one or more users.Meeting information 1732 and identity information 1734 may be maintainedand managed in a wide variety of forms that may vary depending upon aparticular implementation. For example, meeting information 1732 andidentity information 1734 may be maintained as unstructured data in oneor more files, or as structured data in one or more databases. Meetingmanager 1730 may provide one or more APIs that allow IWB appliances1710, 1720 and other client devices to access the functionality providedby meeting manager 1730. Meeting information 1732 may be initiallyobtained and updated from a meeting or calendar system.

B. Meeting Information

FIG. 17C is a block diagram that depicts example contents of meetinginformation 1732 in the form of a table, where each row corresponds to aparticular electronic meeting. In the example depicted in FIG. 17C,meeting information 1732 includes a meeting ID, a meeting name, ameeting location, a date/time for the meeting, participants, and otherinformation. The meeting ID may be any type of information, such as analphanumeric string, that uniquely identifies an electronic meeting. Themeeting name may be, for example, one or more words that describe anelectronic meeting. The meeting location may be any information thatspecifies one or more physical locations for an electronic meeting,including multiple locations. For example, the specified locations mayrefer to meeting or conference rooms, etc. The participants informationspecifies one or more participants for each electronic meeting andparticipants may be indicated by name, user ID, etc. Other informationmay include any type of information about an electronic meeting, forexample, a priority of an electronic meeting, whether the electronicmeeting is public, private, has certain security restrictions, etc.

According to one embodiment, IWB appliance 1710 is registered as a userwith a calendar service to allow IWB appliance 1710 to participate inelectronic meetings in the same manner as users. This may require anadministrator performing authentication, such as OAuth authentication,on behalf of IWB appliance 1710. In addition, other special settings maybe made, for example, to allow IWB appliance 1710 to accept all meetinginvitations by default. Once IWB appliance 1710 has been properlyconfigured to use the calendar service, IWB appliance 1710 may beinvited as an attendee of electronic meetings and given access tomeeting content in the same manner as users.

According to one embodiment, IWB appliance 1710 is configured to supportthe Google Calendar API and a Google account is created and assigned toIWB appliance 1710. The Google account for IWB appliance 1710 may beconfigured to automatically accept meeting invitations. IWB appliance1710 is also configured with the Google Chrome Web browser and anadministrator logs into the Google account created and assigned to IWBappliance 1710 and sets Google Chrome as the default Web browser. Anadministrator user completes OAuth authentication on behalf of IWBappliance 1710 to allow IWB appliance 1710 to access the Google CalendarAPI. In response to a voice command, such as “Show Calendar,” to displaythe Google calendar, IWB appliance 1710 launches the Google Chrome Webbrowser with a Google Calendar URL to allow a user to view scheduledmeetings in the Google Calendar user interface. In response to a voicecommand, such as “Hide Calendar,” to hide the Google calendar, IWBappliance 1710 closes the Google Chrome Web browser. With thisimplementation, meetings may be scheduled with IWB appliance 1710 inseveral ways. First, a user may log into Google Calendar using theaccount created for and assigned to IWB appliance 1710, create a newcalendar entry, invite participants, and specify content items for themeeting. Second, a user may schedule a meeting by sending a meetinginvitation to the account created for and assigned to IWB appliance1710. For example, a user may create a new calendar entry and include,as one of the invitees, the email address of the account created for andassigned to IWB appliance 1710, and specify content items for themeeting.

C. Identity Information

FIG. 17C also depicts example contents of identify information 1734 inthe form of a table, where each row corresponds to a particular person.In the example depicted in FIG. 17B, identity information 1734 includesa user ID, a name, image data, voice data, and other information. Theuser ID may be any data that uniquely identifies a person. Exampleimplementations of a user ID include, without limitation, one or morealphanumeric strings. A person's name may also be an alphanumericstring, e.g., first name, first name and last initial, first name andlast name, etc. Image data may be any type of image data thatcorresponds to the person and that may be used to identify the person.For example, image data may include one or more facial images, one ormore full body images, etc., in various formats. Image data may beencoded in various forms. Image data may also include other data thatrepresents a result of processing one or more images. This type ofinformation may be helpful when attempting to match one or more imageswith image data contained in identity information 1734. For example,image data may include an image signature, the result of processing oneor more images with one or more algorithms, hash functions, etc. Imagedata may be in any format that may vary depending upon a particularimplementation, and embodiments are not limited to any particular formatof image data. Example formats for image data include, withoutlimitation, JPG, TIF, GIF, PNG, RAW, etc.

Voice data may be any type of audio data that corresponds to, and may beused to identify, a person. For example, voice data may represent one ormore words spoken by an individual, such as their name or a spokenphrase. Voice data may be in any format that may vary depending upon aparticular implementation, and embodiments are not limited to anyparticular format of voice data. Example formats for voice data include,without limitation, WMA, WAV, Real Audio (.ra, .ram, .rm), MIDI, Ogg,etc. Voice data may be used alone, or in combination with the imagedata, to identify a speaker.

Other information may include any type of information about a person,including metadata that may be used to identify a person detected withinproximity to an IWB, as described in more detail hereinafter. Examplesof other information include, without limitation, occupation,educational background, work history, accomplishments and achievements,personal interests, physical attributes, such as eye color, hair color,skin color, distinguishing marks, weight, size, etc. Other informationmay also include information about electronic devices, such as mobilephones, portable computers, personal digital assistants, tabletcomputing devices, wearable devices, etc., that belong to a person. Forexample, other information may specify a serial number, MAC address, orother identifying information for a mobile device that belongs to aperson.

Identity information 1734 may be created and maintained in severaldifferent ways that may vary depending upon a particular implementation.For example, identity information may be manually created and maintainedby an administrator of an organization, such as human resourcepersonnel, during the hiring process or new employee orientation.

According to one embodiment, IWB appliance 1710 is configured with thecapability to create and maintain identity information 1734. Thisincludes collecting image and voice data in a passive mode and/or anactive mode. IWB appliance 1710 may be configured to operate in thepassive or active mode, for example, by an administrator, meetingorganizer, etc. IWB appliance 1710 may include controls for selectingthe passive mode, the active mode, or no data collection. The controlsmay also allow a user to select collection of image data, voice data, orboth image data and voice data.

In the passive mode, IWB appliance 1710 acquires facial images ofpersons, such as meeting participants without the participation and/orknowledge of the persons. Facial images may be acquired using one ormore cameras integrated into IWB appliance 1710, such as cameras 1746,or external sensors, as described in more detail hereinafter. IWBappliance 1710 then attempts to associate the acquired facial imageswith particular persons. For example, image recognition application 1752may compare facial images acquired by IWB appliance 1710 to known facialimages from databases, records, social media, etc. This may includeusing meeting participant information. For example, the participants ofa meeting may be determined, and then the acquired facial images may becompared to facial images of meeting participants to associate theacquired facial images with a person.

Voice information may also be used to identify meeting participants,either alone or in combination with facial image information. Forexample, voice recognition application 1754 may compare voicecharacteristics of a person, for whom a facial image has been acquired,to known voice characteristics to determine and/or confirm a person thatcorresponds to the acquired facial images. For example, voiceinformation may be used to confirm the identity of a meeting participantwhen the use of facial image information alone does not specify aminimum confidence threshold. Voice information may also be analyzed todetect clues, such as names of persons, unique words, etc., that may beused to associate an acquired facial image with a person.

Voice data may also be acquired in the passive mode. For example, IWBappliance 1710 may collect voice data via microphones 1742 and attemptto associate the acquired voice data with particular persons. Afterimage data and/or voice data has been acquired via the passive mode andassociated with a particular person, IWB appliance 1710 may updateidentity information 1734 with the new information, for example, byadding a row to the table.

In the active mode, IWB appliance 1710 acquires facial images of personswith their involvement and/or cooperation. For example, IWB appliance1710 may prompt a person, via a visual and/or audible notification, toposition themselves in front of a camera so that a facial image of theperson may be acquired. This may be performed at any time, for example,prior to, during, or after a meeting. The position may be conspicuouslyidentified, for example, by providing prompting and/or live imagefeedback via display 1740. For example, acquired images may be displayedin a particular location of display 1740 and continuously updated toallow a person to properly position themselves for a facial image to beacquired. This may include, for example, displaying a box or frame tovisually indicate an image acquisition area, and displaying live imagesin the box or frame. After acquiring a facial image, IWB appliance 1710may display the acquired image and allow the user to accept the acquiredimage or acquire another image. Once a facial image has been acquiredand confirmed by the user, IWB appliance 1710 may query the user forinformation, such as a name, with which to associate the acquired facialimage. The user may provide the information by typing or speaking theirname, scanning a key card or badge, etc. IWB appliance 1710 may alsoquery the user to speak their name, or one or more words, such as aphrase, to be stored as voice data for the user. After image data and/orvoice data has been acquired via the active mode and associated with aparticular person, IWB appliance 1710 may update identity information1734 with the new information, for example, by adding a row to thetable.

Although meeting information 1732 and identity information 1734 aredepicted as having five entries each, embodiments are not limited tothis example and may include any amount of data. In addition, meetinginformation 1732 and identity information 1734 may be sorted and/orindexed in various ways that may vary depending upon a particularimplementation.

D. Meeting Arrangement

FIG. 17D is a block diagram that depicts a meeting arrangement 1780 thatincludes IWB appliance 1710 (FIG. 17A) and sensors 1784-1792 locatedwithin a room 1782. Room 1782 may be any type of room or defined area,such as a conference room, meeting room, office, presentation area, etc.Although embodiments are described in the context of a room 1782 forpurposes of explanation, embodiments are not limited to this context andare applicable to embodiments without rooms per se.

Sensors 1784-1792 may be any type of sensor for detecting the presenceand/or movement of a human being. Example implementations of sensors1784-1792 include, without limitation, image sensors such as cameras anddepth cameras, voice sensors, audio/video recorders, motion detectors,heat sensors, badge or card readers, scanners, etc. According to oneembodiment, sensors 1784-1792 are capable of generating sensor data thatrepresents the detected condition. For example, an audio sensor maygenerate audio data that represents detected speech and other sounds. Asanother example, an audio/video recorder may generate audio/video datathat represents detected audio and video. As yet another example, amotion detector may generate motion detection data that represents adetected motion.

Sensors 1784-1792 may be implemented by any number of sensors that arelocated in any location with respect to IWB appliance 1710. For example,sensors 1784-1792 may be attached to walls, ceilings, floors, etc.According to one embodiment, sensors 1784-1792 are located in positionswith respect to IWB appliance 1710 to be able to sense the presenceand/or movement of human participants participating in an electronicmeeting via IWB appliance 1710. For example, sensors 1784-1792 may belocated close to where it is expected that participants of theelectronic meeting using IWB appliance 1710 would be physically located.Although embodiments are described herein in the context of sensors1784-1792 that are external to IWB appliance 1710 for purposes ofexplanation, embodiments are not limited to these examples and areapplicable to sensors that are integrated into IWB appliance 1710.

According to one embodiment, IWB appliance 1710 is capable of receivingsensor data generated by sensors 1784-1792, for example, via wired orwireless connections. Sensor data may be provided directly from sensors1784-1792 to IWB appliance 1710, or indirectly via, for example, aserver or sensor data repository. The sensor data may indicate, forexample, movement, the presence of a person, etc.

E. Person Detection, Person Identification and Meeting Start

FIG. 18 is a flow diagram 1800 that depicts an approach for using persondetection and person identification in the context of an electronicmeeting with an IWB appliance. For purposes of explanation, embodimentsare described in the context of a single person being detected by asingle sensor, but embodiments are not limited to this context and areapplicable to any number of persons and sensors. Referring to FIG. 17D,it is presumed that no one is present in room 1782 when person 1794moves from outside room 1782 to inside room 1782, and within sensingproximity of sensor 1790, i.e., within a sensing area 1791 of sensor1790. It is further presumed that sensor 1790 generates sensor data inresponse to detecting person 1794.

In step 1802, an IWB appliance receives, from one or more sensors,sensor data that indicates the physical presence of a person within aspecified distance of the IWB appliance. In the present example, IWBappliance 1710 receives sensor data from sensor 1790 indicating thephysical presence of person 1794 within a specified distance of IWBappliance 1710. The sensor data may be, for example, badge/card readerdata, motion detection sensor data, camera sensor data, and/or audiodata, or any combination thereof, that indicates the presence ormovement of a person. For example, the sensor data may be motiondetection sensor data that indicates movement of a person in thevicinity of IWB appliance 1710. As another example, the sensor data maybe audio data generated in response to a person speaking. As a furtherexample, the sensor data may be scan data generated in response to auser scanning a badge or card. As yet another example, the sensor datamay indicate the presence of an electronic device, such as a mobilephone, that belongs to a particular person. Thus, the sensor data mayindicate that a person is physically located within a specified distanceof IWB appliance 1710.

According to one embodiment, IWB appliance 1710 may be configured tooperate in a low power state, sometimes referred to as a “sleep state,”and transition from the low power state to a higher power state inresponse to receiving the sensor data from sensor 1790. While operatingin the low power state, IWB appliance 1710 continues to execute one ormore processes that provide for the detection of sensor data receivedfrom sensors 1784-1792 while consuming less power than when operating inhigher or more active operational states.

According to one embodiment, upon receiving sensor data indicating thepresence of a person within a specified distance of IWB appliance 1710,IWB appliance 1710 prompts the person for additional information thatmay be used to identify the person. For example, in response toreceiving sensor data indicating the presence of a person, IWB appliance1710 may ask the person to identify themselves by speaking their name.This may be performed, for example, in situations where sensors1784-1792 do not include the capability to capture image or video dataand audio data is therefore needed to identify the person. As anotherexample, IWB appliance 1710 may ask the person to position themselves infront of a camera so that a facial image of the person may be acquired.The position may be conspicuously identified, for example, by providingprompting and/or live feedback via display 1740. For example,video/image input received by one or more of sensors 1784-1792 may bedisplayed and continuously updated on display 1740 to allow the personto properly position themselves to allow a facial image to be acquired.This may include, for example, framing the facial image, usingdirectional indicators, etc. Communications with persons may be madevisually via display 1740, audibly via speakers 1744, or both.

In step 1804, the person is identified using the sensor data. Theparticular steps performed by IWB appliance 1710 to identify a personusing sensor data may vary depending upon the type of sensor data thatwas received. According to one embodiment, IWB appliance 1710 uses thereceived sensor data and identity information 1734 to identify person1794. IWB appliance 1710 may use the local identity information 1734stored on IWB appliance 1710, identity information 1734 obtained frommeeting manager 1730 via one or more requests to meeting manager 1730,or a combination thereof. Identifying a particular person using sensordata may include, for example, comparing one or more portions of thesensor data to image data and/or voice data included in identityinformation 1734. For example, image recognition application 1752 maycompare one or more facial images included in the sensor data to facialimages represented by the image data included in identity information1734 to identify person 1794. As another example, voice recognitionapplication 1754 may compare audio data included in the sensor data tovoice data included in identity information 1734 to identify person1794.

Various types of algorithms and heuristics, for example distancealgorithms, may be used to compare sensor data to image data and/orvoice data included in identity information 1734, and embodiments arenot limited to any particular approach for comparing sensor data to dataincluded in identity information 1734. For example, a distance, such asa Euclidean distance, may be determined between a facial imagerepresented by sensor data and each facial image represented by theimage data included in identity information 1734. The distances arecompared and the facial image represented by the image data included inidentity information 1734 that is closest to the facial imagerepresented by sensor data is determined to be the most likely match.Distance thresholds may also be used. For example, a facial imagerepresented by the image data included in identity information 1734 mustbe less than a threshold distance from the facial image represented bysensor data to be determined as a match. Distance-based approaches mayalso be used to compare audio data represented by sensor data to voicedata in identity information 1734. According to another embodiment, ifthe sensor data is badge or card identification data, such as a numberor code, IWB appliance 1710 contacts a badge or card database to obtainuser information associated with the badge or card identification.

Although embodiments are depicted in the figures and described herein inthe context of performing image and voice recognition using imagerecognition application 1752 and voice recognition application 1754,respectively, on IWB appliance 1710, embodiments are not limited tothese examples and external image and voice recognition services may beused. For example, IWB appliance 1710 may use one or more cloud-basedimage and voice recognition services for processing image and voice datato identify meeting participants. This may include IWB appliance 1710transmitting video and/or audio data to a third party service foranalysis.

Sensor data may be processed prior to being used in comparisons. Forexample, sensor data may be converted into one or more target dataformats prior to being used in comparisons. As another example, one ormore portions of sensor data may be processed, e.g., via one or morealgorithms, to create a face or voice signature, which is then comparedto the image data and voice data included in identity information 1734.Multiple types of sensor data, and sensor data from any number ofsensors, may be used to identify a person. For example, both video/imageand audio data may be used to identify person 1794. Differentcombinations of sensor data may be used depending upon a particularimplementation and/or situation, and embodiments are not limited to anyparticular combination of sensor data. In addition, other informationfrom identity information 1734 may be used to identify person 1794. Forexample, suppose that after a comparison of a facial image included inreceived sensor data to the image data included in identity information1734 that person 1794 is most likely to be User #1 or User #4, but thecomparison scores alone are not definitive. Other information mayinclude attributes of each user, e.g., hair color, eye color, height,weight, distinguishing features or marks, etc., that may be used toconfirm that person 1794 is more likely to be User #1 than User #4. Uponidentification of the person, IWB appliance 1710 may greet the person,for example, via one or more messages displayed on display 1740, one ormore audible greetings via speakers 1744, or both. For example, inresponse to determining that a detected person is “Tom J.,” IWBappliance 1710 may cause the message “Hello Tom J.” to be displayed ondisplay 1740. As another example, IWB appliance 1710 may cause thegreeting “Hello Tom J.” to be spoken/played via speakers 1744. Theprocessing of sensor data, as described herein, may be performed bymeeting manager 1756, one or more services external to IWB appliance1710, or any combination thereof. For example, third party services,such as server-based or cloud-based services may be used to compareimage data or voice data. According to one embodiment, after a person isidentified using sensor data as described herein, an offer is providedfor the identified person to update their corresponding image dataand/or voice data. For example, IWB appliance 1710 may display a visualor audio notification on display 1740 with controls that allow theidentified person to update their corresponding image data and/or voicedata.

In some situations, it may not be possible to identify a person in thevicinity of an IWB appliance. This may occur, for example, if a goodquality facial image of the person cannot be acquired and/or ifbackground noise prohibits acquisition of good quality audio. Accordingto one embodiment, if in step 1804 the person cannot be identified asdescribed herein, the IWB appliance 1710 takes additional actions toattempt to identify the person. The additional actions may include, forexample, querying the user to identify themselves. For example, IWBappliance 1710 may display a message on display 1740, or make an audiorequest via speakers 1744, for the unidentified person to identifythemselves. This may include acquiring another, e.g., better quality,facial image and/or voice data as previously described herein.

In step 1806, a determination is made whether an electronic meetinginvolving IWB appliance 1710 is scheduled at or near the current time.This may be performed using meeting information 1732. For example, IWBappliance 1710 may compare the current time to meeting date/timeinformation for each of the electronic meetings represented in meetinginformation 1732 to determine whether an electronic meeting involvingIWB appliance 1710 is scheduled at or near the current time. Accordingto one embodiment, if the current time is within a specified amount oftime of a scheduled time for a particular electronic meeting, then theparticular electronic meeting is considered to be scheduled at or nearthe current time. The specified amount of time may be configurable, forexample, by an administrative user of IWB appliance 1710, and may bestored as part of configuration data 1736. IWB appliance 1710 may alsoquery a meeting or calendar system to determine whether an electronicmeeting involving IWB appliance 1710 is scheduled at or near the currenttime. For example, IWB appliance 1710 may use an API provided by ameeting or calendar system to obtain meeting information.

If the result of the determination is that an electronic meeting is notscheduled at or near the current time, then in step 1808, an offer ismade to start a new electronic meeting. For example, IWB appliance 1710may provide a visual and/or audible notification that includes aninvitation to start a new electronic meeting, such as “There is nomeeting scheduled at this time. Would you like to start a new meeting?”The identified person may accept or reject the offer to start a newelectronic meeting and if accepted, may create a new meeting via display1740, for example, by selecting graphical user interface objects and/orspecifying information for the new electronic meeting via hand, stylus,or speech.

If, on the other hand, in step 1806 a determination is made that anelectronic meeting is scheduled at or near the current time, then instep 1810, a determination is made to determine whether the identifiedperson is a scheduled participant of the electronic meeting. Forexample, IWB appliance 1710 may consult the participant information forthe electronic meeting from meeting information 1732 to determinewhether the identified person is a scheduled participant of theelectronic meeting.

If, in step 1810 the result of the determination is that the identifiedperson is not a scheduled participant of the electronic meeting, then instep 1812, the identified person is notified of the scheduled electronicmeeting. For example, IWB appliance 1710 may provide a visual and/oraudible notification to the person that an electronic meeting isscheduled. The notification may provide details about the electronicmeeting, such as the name of the electronic meeting, the start time,participants, other information, etc. A calendar may be displayed ondisplay 1740 showing the scheduled meetings for a specified period oftime, e.g., the current day, week, month, etc. Then, in step 1814, anoffer to schedule a new electronic meeting is optionally made. Forexample, IWB appliance 1710 may provide a visual and/or audiblenotification that includes an invitation to schedule a new electronicmeeting. This may include an identification of available meeting times,along with prompts for other information for the electronic meeting,such as a name, participants, agenda, content, etc., for the electronicmeeting. The identified person may accept or reject the offer to start anew electronic meeting and if accepted, may create a new electronicmeeting via display 1740, for example, by selecting graphical userinterface objects and/or specifying information for the new electronicmeeting via hand, stylus, or speech.

If, on the other hand in step 1810, a determination is made that theidentified person is a participant of the electronic meeting, then instep 1816, a conclusion is made that the identified person intends toattend the electronic meeting. In addition, a notification may be madeto welcome the identified person to the electronic meeting. For example,IWB appliance 1710 may provide a visual and/or audible notification theincludes a greeting that welcomes the identified person to theelectronic meeting, such as “Welcome John J. to the Alpha project designreview meeting.” The notification may provide details about theelectronic meeting, such as the name of the electronic meeting, thestart time, participants, other information, etc. Other actions may alsobe performed in response to determining that the identified person is aparticipant of the electronic meeting. For example, as described in moredetail hereinafter, the other actions may include attendance tracking.

In step 1818, IWB appliance 1710 may optionally prepare for and/or startthe electronic meeting. This may include querying the user whether toload and display content for the electronic meeting, such as an agendafor the electronic meeting, and/or one or more content items for theelectronic meeting, such as a presentation, slide show, electronicdocuments, video clips, etc. This may include presenting on display 1740the agenda and a list of content items for the electronic meeting, alongwith graphical user interface controls that allow the identified personto select particular items for viewing and/or editing. IWB appliance1710 may also start a recording of the electronic meeting from which ameeting transcript may be generated. The meeting recording and/or themeeting transcript may be stored in storage 1764.

The aforementioned steps of flow diagram 1800 may be repeated for anynumber of persons, including persons who join an electronic meeting inprogress. The process is also applicable to remote meeting attendees.IWB appliance 1710 may process sensor data collected at other locationsin the same manner as described herein with respect to sensor datacollected from sensors 1784-1792. In addition, IWB appliance 1710 mayprocess image and voice data from meeting data, for example, from anaudio/visual data stream for a videoconferencing session or contentsharing session.

X. Attendance Tracking, Presentation of Content, Meeting Services, andAgenda Extraction

The availability of sensor data to IWB appliances as previouslydescribed herein allows IWB appliances to provide other servicesincluding attendance tracking, presentation of content, meetingservices, and agenda extraction.

A. Attendance Tracking

According to one embodiment, IWB appliances are configured to track theattendance of meeting attendees using sensor data. As depicted in FIG.17B, IWB appliance 1710 maintains attendance tracking information 1766that tracks the attendance of individual participants to particularelectronic meetings. FIG. 19 is a block diagram that depicts an exampleimplementation of attendance tracking information 1766 for an individualnamed “Tom J.” In the example depicted in FIG. 19, each row of the tablecorresponds to a particular electronic meeting attended by “Tom J.” andspecifies information about the electronic meeting, such as a meetingID, a meeting name, a location, a scheduled date/time for the electronicmeeting, and one or more participation times of “Tom J.” Attendancetracking information 1766 may include other information that may varydepending upon a particular implementation, and embodiments are notlimited to the particular attendance tracking information 1766 depictedin FIG. 19.

The participation times are recorded in attendance tracking information1766 by IWB appliance 1710 as a participant enters and leaves anelectronic meeting based upon sensor data from sensors 1784-1792. Forexample, IWB appliance 1710 may record participation times in step 1816of FIG. 18 after the person has been identified in step 1804 anddetermined to be a participant of an electronic meeting scheduled at ornear the current time in step 1810. Participation times may specify theactual times that a participant arrived for and departed an electronicmeeting, based upon sensor data, even if those times are outside thescheduled time for the electronic meeting. For example, suppose that anelectronic meeting is scheduled for 10-11am and a particular participantarrives early for the electronic meeting at 9:55am. The participationstart time may be the particular participant's actual arrival time of9:55am or alternatively, the meeting start time of 10am. Similarly, ifthe particular participant stays beyond the scheduled meeting end timeof 11am to 11:05am, the participation end time may be the particularparticipant's actual departure time of 11:05am or alternatively, themeeting end time of 11am, and the particular approach used may varydepending upon a particular implementation.

As previously described herein, participant identification may includethe use of face recognition, voice recognition, positioning information,scanning information, etc. Participation times may include multipleintervals for a single electronic meeting. For example, Tom J attendedMeeting #4 during two periods, 1:58-2:16pm and 2:20-2:48pm. The gapbetween the two periods, i.e., between 2:16 and 2:20 pm, may represent atime when Tom J left the electronic meeting, for example, to attend toanother issue.

The end of an attendance time period may be determined in various waysthat may vary depending upon a particular implementation. For example,one or more sensors may be located at an entrance to a meeting area sothat the arrival and departure of participants may be explicitlydetermined. As another example, IWB appliance 1710 may detect, viasensor data from sensors 1784-1792, that a participant has endedparticipation in an electronic meeting, for example, by an explicitstatement by the participant that is detected by IWB appliance 1710 viaaudio data. For example, the participant may make the statement,“Goodbye” or “See you later” that is detected by IWB appliance 1710 inaudio data for the electronic meeting. As another example, where aparticular participant was previously detected participating in anelectronic meeting via sensor data from sensors 1784-1792, but is laternot detected via sensor data, it may be presumed that the particularparticipant is no longer participating in the electronic meeting.

The example depicted in FIG. 19 is presented in the context of a singleparticipant for purposes of explanation only, and attendance trackinginformation 1766 may include meeting tracking information for any numberof participants. For example, attendance tracking information 1766 mayinclude a separate table for each participant. Embodiments are notlimited to the example depicted in FIG. 19, and attendance trackinginformation 1766 may different information depending upon a particularimplementation. Attendance tracking information 1766 may be shared withand/or aggregated across other devices. For example, IWB appliance 1710may share and/or aggregate attendance tracking information 1766 with IWBappliance 1720, meeting manager 1730, etc.

FIG. 20 is a flow diagram 2000 that depicts an approach for performingmeeting attendance tracking for an IWB appliance according to anembodiment. In step 2002, an IWB appliance receives, from one or moresensors, first sensor data that indicates the physical presence of aperson within a specified distance of an IWB appliance. In the presentexample, IWB appliance 1710 receives first sensor data from sensor 1790indicating the physical presence of person 1794 within a specifieddistance of IWB appliance 1710.

In step 2004, the person is identified using the first sensor data aspreviously described herein. In step 2006, a determination is madewhether an electronic meeting involving IWB appliance 1710 is scheduledat or near the current time, as previously described herein. If theresult of the determination is that an electronic meeting is notscheduled at or near the current time, then in step 2008, an offer ismade to start a new electronic meeting, as previously described herein.If, on the other hand, in step 2006 a determination is made that anelectronic meeting is scheduled at or near the current time, then instep 2010, a determination is made to determine whether the identifiedperson is a scheduled participant of the electronic meeting, aspreviously described herein. If, in step 2010 the result of thedetermination is that the identified person is not a scheduledparticipant of the electronic meeting, then in step 2012, the identifiedperson is notified of the scheduled electronic meeting, as previouslydescribed herein.

If, on the other hand in step 2010, a determination is made that theidentified person is a participant of the electronic meeting, then adetermination is made that the identified person has joined theelectronic meeting and in step 2016, first participation data isgenerated to specify a time at which the identified person joined theelectronic meeting. For example, IWB appliance 1710 may updateattendance tracking information 1766 with a time, i.e., a start time, tospecify the identified person's participation in the electronic meeting.Sometime later, in step 2018, second sensor data is received thatindicates that the identified person is no longer attending theelectronic meeting. The second sensor data may explicitly indicate thatthe identified person has departed room 1782. For example, one or moresensors, e.g., badge/card readers, cameras, etc., may detect theidentified person leaving room 1782. In this manner, the second sensordata explicitly indicates that the identified person is no longer withthe specified distance of IWB appliance 1710. According to anotherembodiment, the second sensor data may no longer detect the presence ofthe identified person, and a corresponding determination is made thatthe identified person is no longer within the specified distance of IWBappliance 1710.

In response to receiving the second sensor data that indicates that theidentified person is no longer attending the electronic meeting, in step2020, second participation data is generated to specify a time at whichthe identified person ended participation in the electronic meeting. Forexample, IWB appliance 1710 may update attendance tracking information1766 with a time, i.e., an end time, to specify the identified person'sparticipation in the electronic meeting.

B. Presentation of Content

As previously described herein, e.g., step 1818 of FIG. 18, IWBappliance 1710 may optionally prepare for and/or start a meeting, whichmay include obtaining and/or displaying content, referred to herein as“content items,” for the meeting. Content items may include any type ofcontent that may vary depending upon a particular implementation.Examples of content items include, without limitation, an agenda for themeeting, and/or one or more content items for the meeting, such aspresentations, slide shows, electronic documents, video clips, etc.

IWB appliance 1710 may display representations of content items toprovide a visual indication of the content items. For example, IWBappliance 1710 may display icons, symbols, images, such as thumbnailimages, etc., to visually indicate content items that may be displayedfor a meeting. IWB appliance 1710 may also display visual controls, inthe form of graphical user interface objects, that allow a user toselect one or more content items to be displayed for a meeting. A usermay select a visual control by touching display 1740 with their finger,using a stylus, etc. According to one embodiment, IWB appliance 1710provides voice command support for displaying content items. This mayinclude voice commands for selecting and displaying content items. Forexample, a user may speak the command, “IWB Command: display Acornpresentation,” and in response to the command, IWB appliance 1710displays the Acorn presentation on display 1740. In this example, thespoken words “IWB Command” serve as a marker or flag in audio data tohelp IWB appliance 1710 recognize voice commands.

As another alternative, IWB appliance 1710 may automatically displaycontent items for a meeting and may arrange the content items, orrepresentations thereof, using tiling, separate windows, etc., tovisually organize the content items in a manner that allows a user torecognize that multiple content items are available without necessarilybeing able to simultaneously view the entirety of all of the contentitems. IWB appliance 1710 may use intelligence to group content itemshaving similar content. For example, content items that are related to afirst subject are grouped together as a first set of content items anddisplayed in a first region of display 1740. Content items that arerelated to a second subject are grouped together as a second set ofcontent items and displayed in a second region of display 1740 that isdifferent than the first region, separate from the first set of contentitems. This may be implemented by displaying the actual content items,or representations thereof, as previously described herein. Contentitems, or representations thereof, may also be visually organized ondisplay 1740 by date.

Content items may be obtained by IWB appliance 1710 from a wide varietyof sources and/or locations that may vary depending upon a particularimplementation and embodiments are not limited to any particular source.For example, IWB appliance 1710 may obtain content items locally fromstorage 1764, from servers, databases, etc., that are remote withrespect to IWB appliance 1710, or any combination thereof. The sourceand/or location of content items may be specified by configuration data1736. Any number of content items of the same or different types may bedisplayed for a meeting by IWB appliance 1710. According to oneembodiment, IWB appliance 1710 is configured to satisfy anyauthentication and/or access requirements required to obtain contentitems. For example, IWB appliance 1710 may supply access credentialsrequired to retrieve content items from a repository or database. Inaddition, IWB appliance 1710 may be configured to generate requests forcontent items that satisfy an API of a server, database, etc. Forexample, IWB appliance 1710 may generate and transmit an SQL query to adatabase management system to request one or more content items.

The determination of particular content items for a meeting may be madein different ways that may vary depending upon a particularimplementation, and embodiments are not limited to any particularapproach. For example, content items for a meeting may be specified bymeeting information 1732. As another example, content items may bespecified by calendar data that defines a meeting. For example, someelectronic meeting/calendar systems allow content items to be specifiedwhen a meeting is created and the corresponding meeting invitations mayidentify or include the content items, e.g., as attachments, and/orinclude a link to content items. Thus, in situations where IWB appliance1710 is an invited attendee, IWB appliance 1710 is able to determine andobtain the content items for the meeting based upon the meetinginvitation and/or calendar data. This may include obtaining contentitems from a meeting invitation, from a calendaring system, or anotherlocation referenced by a meeting invitation.

For example, suppose that User A creates an electronic meeting in anelectronic meeting system and invites User B and User C to the meeting.User A intends on conducting the electronic meeting in room 1782 usingIWB appliance 1710, so User A also invites IWB appliance 1710 to themeeting. In this example, User A is the meeting organizer, but this isnot required and users may create meetings on behalf of other users.Suppose further that User A specifies that Content Items #1 and #2 willbe used for the meeting. In response to the creation of the meeting, theelectronic meeting system sends an invitation to User B, User C, and IWBappliance 1710, for example, via email. The meeting invitation specifiesdetails for the electronic meeting, such as date, time, location, etc.,and also specifies that Content Items #1 and #2 will be used for theelectronic meeting. Content Items #1 and #2 may be included asattachments to the meeting invitation, and/or may be specified by themeeting invitation. For example, the meeting invitation may identifyContent Items #1 and #2, include a link to Content Items #1 and #2,e.g., to local storage 1764 or remote service, such as a cloud-basedservice, etc., or include Content Items #1 and #2 as attachments to themeeting invitation. Thus, in this example, User A, User B, User C, andIWB appliance 170 all have access to Content Items #1 and #2 via themeeting invitation.

Content items may be processed by IWB appliances in vary ways tofacilitate processing by IWB appliances. For example, content items maybe deconstructed from their original file formats to extract text andother content contained therein. Extracted content items may be storedas structured data, for example, in a database management system, orstored in meeting information 1732. As another example, the format ofcontent items may be changed to accommodate additional processing, suchas searching, etc.

C. Meeting Services

According to one embodiment, IWB appliances are configured to providevarious meeting services. As used herein, the term “meeting service”refers to a service that has a correspondence to an electronic meetingconducted via an IWB appliance. Examples of meeting services include,without limitation, transportation, lodging, food, entertainment,technical support, and information requests, and embodiments are notlimited to any particular types of meeting services. The correspondencebetween a meeting service and an electronic meeting may include any typeof relationship between an electronic meeting and a meeting service. Forexample, a meeting service may be requested prior to, during, or afteran electronic meeting, and considered to have a correspondence with theelectronic meeting.

According to one embodiment, an IWB appliance is configured to providevoice command support for requesting meeting services. Once a IWBappliance has been activated, either manually or by detecting thepresence of one or more persons, IWB appliance may begin monitoringaudio data to detect voice commands pertaining to meeting services. Forexample, a user may request a meeting service by speaking the command,“IWB Command: meeting service, order two extra-large pepperoni pizzas.”IWB appliance 1710 detects or recognizes the command in audio data andfulfills the requested meeting service. As another example, a user mayspeak the command, “IWB Command: taxi service.” IWB appliance 1710 againrecognizes the command in audio data and fulfills the requested meetingservice. Meeting services may be requested at any time with respect toan electronic meeting, including before, during, and after an electronicmeeting. For example, meeting services that are to be used during anelectronic meeting may be requested before or during an electronicmeeting. As another example, a car service that is to be used after anelectronic meeting may be requested prior to, during, or after anelectronic meeting. According to one embodiment, IWB appliances providesupport for pre-ordering of meeting services. For example, suppose thatan electronic meeting is schedule for 1-4pm on a particular day. Supposefurther that the meeting participants want to have a particular meetingservice, such as food, etc., provided at 3pm. A person may speak thecommand, “IWB Command: meeting service, order two extra-large pepperonipizzas, deliver at 3pm.” As another example, a person may speak thecommand, “IWB Command: meeting service, taxi at 4:15pm.” In bothexamples, IWB appliance 1710 detects the spoken command andautomatically fulfills the meeting service requests at the specifiedtimes.

IWB appliance 1710 may be configured to fulfill meeting service requestsusing specific services. For example, a user, such as an administrativeuser, may specify that IWB appliance 1710 uses a particular pizza parlorto fulfill orders for pizza, or a particular taxi service to fulfillrequests for taxi service, and this information is stored inconfiguration data 1736. IWB appliances may be configured to generateand transmit requests for meeting services that comply with anyrequirements of third party services used to fulfill the requests, suchas APIs, security, etc. For example, IWB appliance 1710 may beconfigured to generate and transmit a request for taxi service to a Webapplication for the taxi service, and the request satisfies anyrequirements of the Web application. IWB appliances may be configured toprovide visual confirmation of requests for meeting services. Forexample, in response to detecting a command for a meeting service, IWBappliance 1710 may display a text message confirming the request for themeeting service. IWB appliance 1710 may also display options for arequested meeting service and/or request additional information fromusers. For example, suppose that IWB appliance 1710 is not configuredwith a particular service to fulfill requests for taxi services. In thissituation, IWB appliance 1710 may display several car service optionsvia display 1740 along with graphical controls for selecting aparticular car service option. A user may select a graphical controlthat corresponds to a desired car service or speak a voice command toselect the desired car service. In response to a user selecting aparticular car service, IWB appliance 1710 may query the user whetherthe particular car service should be established as the default servicefor fulfilling future car service requests. In response to the userconfirming this query, IWB appliance 1710 may update configuration data1736 to specify that the particular car service should be used forfuture car service requests. IWB appliance 1710 may also provide statusupdates for requested meeting services, for example, via display 1740.

Meeting services may also be requested via an IWB appliance in otherways besides voice recognition. This may include using a live operator,a chat bot, an avatar, or any combination thereof. For example, IWBappliance 1710 may be configured to provide an audio or an audio/videoconnection with a live operator to allow a user to request meetingservices from the live operator. Similarly, IWB appliance 1710 mayestablish a connection with a chat bot or an avatar, with whom users maymake requests for meeting services.

Requests for meeting services involving information requests may includerequests for information within an organization. For example, a user mayrequest information from prior meetings, such as agendas, content items,action items, meeting transcripts, etc. Information requests may beprovided to internal database management systems, business systems, orother enterprise systems for fulfillment. Results provides by theseentities may be displayed on the display of an IWB appliance.

D. Agenda Extraction

According to one embodiment, IWB appliances are configured to providevarious agenda-related services that may vary depending upon aparticular implementation. For example, when content items for anelectronic meeting include an agenda, an IWB appliance may display theagenda, or a representation thereof, in a conspicuous manner on display1740 to call attention to the agenda.

IWB appliances may obtain agenda information from various sources. Forexample, IWB appliances may retrieve agendas attached to, or identifiedby, meeting invitations, calendar entries, etc. As another example, IWBappliances may retrieve agenda information by conducting keywordsearches of meeting data or calendar data to search for agenda-relateddocuments. For example, IWB appliance 1710 may search for the word“Agenda” in calendar/meeting entries, notes, description fields, etc.,to locate agenda information. According to another embodiment, IWBappliances are configured to search for keywords contained in contentitems to search for agenda-related information. For example, IWBappliance 1710 may search for the word “Agenda” in content items, suchas presentation documents, word processing documents, spreadsheetdocuments, etc., to identify meeting agenda information containedtherein. This allows an IWB appliance to automatically detect and use anagenda, without a user having to manually specify agenda items. Once anagenda has been identified and retrieved, the agenda may be used withthe other functionality described herein for agendas.

XI. Learning

According to one embodiment, IWB appliances are configured with theability to perform learning based upon prior IWB appliance usage in thecontext of electronic meetings. This includes collecting IWB applianceusage information about the usage of IWB appliances before, during andafter electronic meetings, analyzing the collected IWB appliance usageinformation to determine usage patterns, and taking one or more actionsbased upon the determined usage patterns.

IWB appliance usage information may include a wide variety ofinformation that may vary depending upon a particular implementation,and embodiments are not limited to any particular type of information.Examples of IWB appliance usage information include, without limitation,IWB appliance settings, meeting information, such as meeting dates andtimes, meeting types, meeting organizers, attendees and content itemsused during electronic meetings, meeting services used before, during,and after electronic meetings, and control system settings for systemsexternal to IWB appliances.

FIG. 21 is a flow diagram 2100 that depicts an approach for improvingthe use of IWB appliances in the context of electronic meetings basedupon IWB appliance usage information. In step 2102, IWB appliance usageinformation is collected. For example, IWB appliance 1710 may collectand store IWB appliance usage information 1768 (FIG. 17B) in the contextof electronic meetings conducted via IWB appliance 1710.

FIG. 22 is a block diagram that depicts example IWB appliance usageinformation 1768 that includes IWB appliance settings 2210, meetinginformation 2220, meeting services 2230, and control system settings2240. IWB appliance settings 2210 specify IWB appliance settings usedduring electronic meetings including, without limitation, displaysettings (brightness, color, contrast, etc.), camera and microphonesettings (volume, sensitivity, etc.), operating mode (local,local/remote), input/output settings, other configuration settings.Meeting information 2220 may specify meeting name, meeting type, meetinglocation, meeting dates and times, meeting organizers, attendees andcontent items used during electronic meetings. Meeting services 2230 mayspecify meeting services used before, during and after electronicmeetings. Control system settings 2240 include settings for systemsexternal to IWB appliances, such as temperature and lighting controls,video and audio system controls, etc.

According to one embodiment, IWB appliance usage information 1768 isgenerated based upon prior electronic meetings that have been conducted,and is then dynamically updated over time as additional electronicmeetings are conducted. Meeting manager 1756 may examine and extractmeeting information from meeting information 1732, attendance trackinginformation 1766, electronic calendaring and meeting systems, electronicmeeting transcripts and notes, and use the extracted meeting informationto generate meeting information 2220. IWB appliance usage information1768 may be generated based upon any number of prior electronic meetingsand, as discussed in more detail hereinafter, may provide better resultsfor a greater number of prior electronic meetings and in particular,multiple occurrences of each of a plurality of electronic meetings. Forexample, multiple occurrences of a department meeting, a projectmeeting, an organization meeting, etc.

According to one embodiment, IWB appliance usage information 1768specifies a plurality of IWB appliance usage values, where each IWBappliance usage value specifies an IWB appliance setting, meetinginformation, a meeting service, or a control system setting that wasused for a prior electronic meeting. For example, an IWB appliance usagevalue may specify an IWB appliance setting in the form of a brightnesssetting of display 1740 that was used for a prior electronic meeting. Asanother example, an IWB appliance usage value may specify meetinginformation in the form of the organizer or a participant of a priorelectronic meeting. As a further example, an IWB appliance usage valuemay specify a meeting service used in conjunction with, i.e., before,during or after, a prior electronic meeting. As another example, an IWBappliance usage value may specify a control system setting in the formof a lighting or temperature setting used for a prior electronicmeeting.

In step 2104, the IWB appliance usage information is analyzed todetermine usage patterns for the prior electronic meetings. IWBappliance usage information 1768 may be analyzed by IWB appliance 1710or external to IWB appliance 1710. For example, one or more processes onIWB appliance 1710, such as meeting manager 1756, may analyze IWBappliance usage information 1768 to identify usage patterns for priorelectronic meetings. This may include the use of any number and types ofheuristics, algorithms, machine learning, artificial intelligence, etc.As another example, IWB appliance 1710 may provide one or more portions(or all) of IWB appliance usage information 1768 to one or moreprocesses external to IWB appliance 1710, such as server-based,network-based, cloud-based services, including without limitation,artificial intelligence service 110, for analysis to identify usagepatterns for prior electronic meetings.

Usage patterns may take different forms that may be specific toparticular IWB appliance usage information. According to one embodiment,a usage pattern may be determined when the same IWB appliance usagevalue was used for two or more prior electronic meetings. The same IWBappliance usage value may apply to any type of IWB appliance usageinformation. For example, two or more prior electronic meetingsinvolving IWB appliance 1710 may use one or more of the same IWBappliance settings, meeting days, times, location, attendees or contentitems, use the same meeting service(s), and/or the same control systemsettings. In addition, the two or more prior electronic meetings do notneed to have been contiguous in time.

Consider the following example. Suppose that a user, User A, conductstwo code design review meetings for the Pluto project using IWBappliance 1710. Suppose further that at least some of the IWB applianceusage information is the same across both meetings. For example, themeetings may have been conducted with the same IWB appliance settings,on the same day of the week, in the same location, with participants incommon, with one or more content items in common, one or more meetingservices in common, and/or one or more control system settings incommon. In this example, one or more usage patterns are determined basedupon the IWB appliance usage information that is common across the twocode design review meetings.

In step 2106, one or more actions for a new electronic meeting using theIWB appliance are performed in response to the determined usagepatterns. The one or more actions may include a wide variety of actionsthat may vary depending upon a particular implementation, andembodiments are not limited to any particular actions. According to oneembodiment, when another occurrence of the electronic meeting involvingIWB appliance 1710 is created and/or scheduled via IWB appliance 1710,one or more of the IWB appliance usage values from the identified usagepatterns are used for the new electronic meeting.

For example, one or more IWB appliance settings that were used formultiple prior electronic meetings involving IWB appliance 1710 may beused for the new electronic meeting. As another example, recurring date,such as every Friday or the third Wednesday of each month, orparticipants of the prior electronic meetings may be automatically usedfor the new electronic meeting. As a further example, one or morecontent items, meeting services, or control system settings that wereused in the prior electronic meetings may be automatically used for thenew electronic meeting. The automatic inclusion of prior IWB applianceusage values for the new electronic meeting may be accomplished in manydifferent ways that may vary depending upon a particular implementationand embodiments are not limited to any particular way. For example, whenthe new electronic meeting is created, the IWB appliance usage valuesfrom the prior electronic meetings may be designated as default valuesfor the new electronic meeting and displayed to a user via display 1740.Graphical user interface controls may be displayed with the defaultvalues to allow the user to accept or change the default values. Forexample, when a user creates the new meeting, the participants of theprior electronic meetings may be listed as default attendees for the newelectronic meeting and graphical user interface controls displayedadjacent each default attendee to allow the user to accept or rejecteach of the default attendees. Selecting a graphical user interfacecontrol to accept a particular default attendee causes the particulardefault attendee to be confirmed as an intended participant of the newelectronic meeting and may cause, for example, a meeting invitation tobe generated and transmitted to the particular default attendee, e.g.,using the email address of the default attendee. As an alternative toautomatically including IWB appliance usage information values for newelectronic meetings, IWB appliance usage information values that areidentified in the usage patterns may be provided as suggestions tousers, and users are allowed to confirm or reject the suggestions. Forexample, participants that participated in the prior electronic meetingsmay be presented on display 1740 of IWB appliance 1710, or othercalendar/meeting scheduling processes, as suggested participants, alongwith graphical user interface controls that allow a user to confirm orreject each of the suggested participants. IWB appliance usage valuesmay be used as default or suggested values on a value-by-value basis.For example, IWB appliance settings values from prior electronicmeetings may be used as default values for the new electronic meeting,while meeting attendees from the prior electronic meetings may be usedas suggested attendees for the new electronic meeting.

The use of IWB appliance settings from prior electronic meetings is notlimited to the context of creating a new meeting, and may be used inother ways. According to an embodiment, at any time during or after anelectronic meeting, an IWB appliance may, upon determining usagepatterns for prior electronic meetings, suggest a new electronic meetingbased upon the usage patterns for the prior electronic meetings. Forexample, upon determining a usage pattern that a particular electronicmeeting using IWB appliance 1710 occurs on every Friday morning from10-11am and includes particular content items, such as a requirementsspecification, IWB appliance 1710 may display a message asking whetherto automatically create a next occurrence of the electronic meeting forthe next Friday at 10-11 am, and including the particular content items,i.e., the requirements specification. IWB appliance 1710 may includegraphical user interface controls that allow a user to confirm or rejectthe proposed new electronic meeting, and may also include graphical userinterface controls for automatically creating multiple recurringmeetings every Friday from 10-11am, with the particular content items.

The learning techniques described herein provide a novel and favorableuser experience by providing default and/or suggested values for newelectronic meetings using IWB appliances based upon prior electronicmeetings conducted with the IWB appliances. The learning techniques alsoreduce the amount of computational and storage resources consumed by IWBappliances.

XII. Implementation Examples

Although the flow diagrams of the present application depict aparticular set of steps in a particular order, other implementations mayuse fewer or more steps, in the same or different order, than thosedepicted in the figures.

According to one embodiment, the techniques described herein areimplemented by one or more special-purpose computing devices. Thespecial-purpose computing devices may be hard-wired to perform thetechniques, or may include digital electronic devices such as one ormore application-specific integrated circuits (ASICs) or fieldprogrammable gate arrays (FPGAs) that are persistently programmed toperform the techniques, or may include one or more general purposehardware processors programmed to perform the techniques pursuant toprogram instructions in firmware, memory, other storage, or acombination. Such special-purpose computing devices may also combinecustom hard-wired logic, ASICs, or FPGAs with custom programming toaccomplish the techniques. The special-purpose computing devices may bedesktop computer systems, portable computer systems, handheld devices,networking devices or any other device that incorporates hard-wiredand/or program logic to implement the techniques.

FIG. 23 is a block diagram that depicts an example computer system 2300upon which embodiments may be implemented. Computer system 2300 includesa bus 2302 or other communication mechanism for communicatinginformation, and a processor 2304 coupled with bus 2302 for processinginformation. Computer system 2300 also includes a main memory 2306, suchas a random-access memory (RAM) or other dynamic storage device, coupledto bus 2302 for storing information and instructions to be executed byprocessor 2304. Main memory 2306 also may be used for storing temporaryvariables or other intermediate information during execution ofinstructions to be executed by processor 2304. Computer system 2300further includes a read only memory (ROM) 2308 or other static storagedevice coupled to bus 2302 for storing static information andinstructions for processor 2304. A storage device 2310, such as amagnetic disk or optical disk, is provided and coupled to bus 2302 forstoring information and instructions.

Computer system 2300 may be coupled via bus 2302 to a display 2312, suchas a cathode ray tube (CRT), for displaying information to a computeruser. Although bus 2302 is illustrated as a single bus, bus 2302 maycomprise one or more buses. For example, bus 2302 may include withoutlimitation a control bus by which processor 2304 controls other deviceswithin computer system 2300, an address bus by which processor 2304specifies memory locations of instructions for execution, or any othertype of bus for transferring data or signals between components ofcomputer system 2300.

An input device 2314, including alphanumeric and other keys, is coupledto bus 2302 for communicating information and command selections toprocessor 2304. Another type of user input device is cursor control2316, such as a mouse, a trackball, or cursor direction keys forcommunicating direction information and command selections to processor2304 and for controlling cursor movement on display 2312. This inputdevice typically has two degrees of freedom in two axes, a first axis(e.g., x) and a second axis (e.g., y), that allows the device to specifypositions in a plane.

Computer system 2300 may implement the techniques described herein usingcustomized hard-wired logic, one or more ASICs or FPGAs, firmware and/orprogram logic or computer software which, in combination with thecomputer system, causes or programs computer system 2300 to be aspecial-purpose machine. According to one embodiment, those techniquesare performed by computer system 2300 in response to processor 2304processing instructions stored in main memory 2306. Such instructionsmay be read into main memory 2306 from another non-transitorycomputer-readable medium, such as storage device 2310. Processing of theinstructions contained in main memory 2306 by processor 2304 causesperformance of the functionality described herein. In alternativeembodiments, hard-wired circuitry may be used in place of or incombination with software instructions to implement the embodiments.Thus, embodiments are not limited to any specific combination ofhardware circuitry and software.

The term “non-transitory computer-readable medium” as used herein refersto any non-transitory medium that participates in providing data thatcauses a computer to operate in a specific manner. In an embodimentimplemented using computer system 2300, various computer-readable mediaare involved, for example, in providing instructions to processor 2304for execution. Such media may take many forms, including but not limitedto, non-volatile and volatile non-transitory media. Non-volatilenon-transitory media includes, for example, optical or magnetic disks,such as storage device 2310. Volatile non-transitory media includesdynamic memory, such as main memory 2306. Common forms of non-transitorycomputer-readable media include, without limitation, a floppy disk, aflexible disk, hard disk, magnetic tape, or any other magnetic medium, aCD-ROM, any other optical medium, a RAM, a PROM, and EPROM, aFLASH-EPROM, any other memory chip, memory cartridge or memory stick, orany other medium from which a computer can read.

Various forms of non-transitory computer-readable media may be involvedin storing instructions for processing by processor 2304. For example,the instructions may initially be stored on a storage medium of a remotecomputer and transmitted to computer system 2300 via one or morecommunications links. Bus 2302 carries the data to main memory 2306,from which processor 2304 retrieves and processes the instructions. Theinstructions received by main memory 2306 may optionally be stored onstorage device 2310 either before or after processing by processor 2304.

Computer system 2300 also includes a communication interface 2318coupled to bus 2302. Communication interface 2318 provides acommunications coupling to a network link 2320 that is connected to alocal network 2322. For example, communication interface 2318 may be amodem to provide a data communication connection to a telephone line. Asanother example, communication interface 2318 may be a local areanetwork (LAN) card to provide a data communication connection to acompatible LAN. Wireless links may also be implemented. In any suchimplementation, communication interface 2318 sends and receiveselectrical, electromagnetic or optical signals that carry digital datastreams representing various types of information.

Network link 2320 typically provides data communication through one ormore networks to other data devices. For example, network link 2320 mayprovide a connection through local network 2322 to a host computer 2324or to data equipment operated by an Internet Service Provider (ISP)2326. ISP 2326 in turn provides data communication services through theworld wide packet data communication network now commonly referred to asthe “Internet” 2328. Local network 2322 and Internet 2328 both useelectrical, electromagnetic or optical signals that carry digital datastreams.

Computer system 2300 can send messages and receive data, includingprogram code, through the network(s), network link 2320 andcommunication interface 2318. In the Internet example, a server 2330might transmit a requested code for an application program throughInternet 2328, ISP 2326, local network 2322 and communication interface2318. The received code may be processed by processor 2304 as it isreceived, and/or stored in storage device 2310, or other non-volatilestorage for later execution.

In the foregoing specification, embodiments have been described withreference to numerous specific details that may vary from implementationto implementation. Thus, the sole and exclusive indicator of what is,and is intended by the applicants to be, the invention is the set ofclaims that issue from this application, in the specific form in whichsuch claims issue, including any subsequent correction. Hence, nolimitation, element, property, feature, advantage or attribute that isnot expressly recited in a claim should limit the scope of such claim inany way. The specification and drawings are, accordingly, to be regardedin an illustrative rather than a restrictive sense.

What is claimed is:
 1. An interactive whiteboard (IWB) appliancecomprising: a display; a communications interface; one or moreprocessors; and one or more memories storing instructions which, whenprocessed by the one or more processors, cause: receiving, via thecommunications interface, from one or more sensors, sensor data thatindicates a detected presence of a person, identifying, based upon thesensor data that indicates a detected presence of a person and identityinformation that specifies one or more of image data or voice data foreach of a plurality of persons, a particular person from the pluralityof persons that corresponds to the sensor data that indicates a detectedpresence of a person, determining, based upon meeting information and acurrent time, whether an electronic meeting involving the IWB applianceis scheduled at or around the current time, in response to determining,based upon the meeting information and the current time, that anelectronic meeting involving the IWB appliance is scheduled at or aroundthe current time, determining, based upon the meeting information,whether the particular person is a scheduled participant of theparticular electronic meeting involving the IWB appliance that isscheduled at or around the current time, in response to determining,based upon the meeting information, that the particular person is ascheduled participant of the particular electronic meeting involving theIWB appliance that is scheduled at or around the current time, generateand display on the display of the IWB appliance one or morenotifications pertaining to the particular electronic meeting.
 2. TheIWB appliance of claim 1, wherein identifying, based upon the sensordata that indicates a detected presence of a person and identityinformation that specifies one or more of image data or voice data foreach of a plurality of persons, a particular person from the pluralityof persons that corresponds to the sensor data that indicates a detectedpresence of a person, includes one or more of matching a facial imagefor the particular person in the identity information to a facial imagerepresented in the sensor data, or matching voice data for theparticular person in the identity information to voice data representedin the sensor data.
 3. The IWB appliance of claim 1, wherein the one ormore sensors include one or more image sensors, voice sensors, orscanners.
 4. The IWB appliance of claim 1, wherein the one or morememories store additional instructions which, when processed by the oneor more processors, cause in response to determining, based upon themeeting information that the particular person is not a participant ofthe particular electronic meeting involving the IWB appliance that isscheduled at or around the current time, offering to the particularperson, via the display of the IWB appliance, to schedule a newelectronic meeting.
 5. The IWB appliance of claim 1, wherein the one ormore memories store additional instructions which, when processed by theone or more processors, cause in response to determining, based upon themeeting information and the current time, that an electronic meetinginvolving the IWB appliance is not scheduled at or around the currenttime, offering to the particular person, via the display of the IWBappliance, to schedule a new electronic meeting.
 6. The IWB appliance ofclaim 1, wherein the one or more memories store additional instructionswhich, when processed by the one or more processors, cause in responseto determining, based upon the meeting information and the current time,that an electronic meeting involving the IWB appliance is not scheduledat or around the current time, offering to the particular person, viathe display of the IWB appliance, to schedule a new electronic meeting.7. The IWB appliance of claim 1, wherein the one or more memories storeadditional instructions which, when processed by the one or moreprocessors, cause the IWB appliance to acquire one or more facial imagesof, or one or more words spoken by, a particular person from theplurality of persons.
 8. The IWB appliance of claim 7, wherein the IWBappliance acquires the one or more facial images of, or one or morewords spoken by, the particular person from the plurality of personswithout involvement of the particular person.
 9. The IWB appliance ofclaim 1, wherein the one or more notifications pertaining to theparticular electronic meeting include one or more of an announcement ofthe meeting or a welcome to the meeting.
 10. The IWB appliance of claim1, wherein the one or more memories store additional instructions which,when processed by the one or more processors, cause the IWB appliance todisplay content for the meeting on the display.
 11. One or morenon-transitory computer-readable media storing instructions which, whenprocessed by the one or more processors, cause: receiving, by aninteractive whiteboard (IWB) appliance via a communications interface,from one or more sensors, sensor data that indicates a detected presenceof a person, identifying, based upon the sensor data that indicates adetected presence of a person and identity information that specifiesone or more of image data or voice data for each of a plurality ofpersons, a particular person from the plurality of persons thatcorresponds to the sensor data that indicates a detected presence of aperson, determining, based upon meeting information and a current time,whether an electronic meeting involving the IWB appliance is scheduledat or around the current time, in response to determining, based uponthe meeting information and the current time, that an electronic meetinginvolving the IWB appliance is scheduled at or around the current time,determining, based upon the meeting information, whether the particularperson is a scheduled participant of the particular electronic meetinginvolving the IWB appliance that is scheduled at or around the currenttime, in response to determining, based upon the meeting information,that the particular person is a scheduled participant of the particularelectronic meeting involving the IWB appliance that is scheduled at oraround the current time, generate and display on the display of the IWBappliance one or more notifications pertaining to the particularelectronic meeting.
 12. The one or more non-transitory computer-readablemedia of claim 11, wherein identifying, based upon the sensor data thatindicates a detected presence of a person and identity information thatspecifies one or more of image data or voice data for each of aplurality of persons, a particular person from the plurality of personsthat corresponds to the sensor data that indicates a detected presenceof a person, includes one or more of matching a facial image for theparticular person in the identity information to a facial imagerepresented in the sensor data, or matching voice data for theparticular person in the identity information to voice data representedin the sensor data.
 13. The one or more non-transitory computer-readablemedia of claim 11, wherein the one or more sensors include one or moreimage sensors, voice sensors, or scanners.
 14. The one or morenon-transitory computer-readable media of claim 11, wherein the one ormore memories store additional instructions which, when processed by theone or more processors, cause in response to determining, based upon themeeting information that the particular person is not a participant ofthe particular electronic meeting involving the IWB appliance that isscheduled at or around the current time, offering to the particularperson, via the display of the IWB appliance, to schedule a newelectronic meeting.
 15. The one or more non-transitory computer-readablemedia of claim 11, wherein the one or more memories store additionalinstructions which, when processed by the one or more processors, causein response to determining, based upon the meeting information and thecurrent time, that an electronic meeting involving the IWB appliance isnot scheduled at or around the current time, offering to the particularperson, via the display of the IWB appliance, to schedule a newelectronic meeting.
 16. The one or more non-transitory computer-readablemedia of claim 11, wherein the one or more memories store additionalinstructions which, when processed by the one or more processors, causein response to determining, based upon the meeting information and thecurrent time, that an electronic meeting involving the IWB appliance isnot scheduled at or around the current time, offering to the particularperson, via the display of the IWB appliance, to schedule a newelectronic meeting.
 17. The one or more non-transitory computer-readablemedia of claim 11, wherein the one or more memories store additionalinstructions which, when processed by the one or more processors, causethe IWB appliance to acquire one or more facial images of, or one ormore words spoken by, a particular person from the plurality of persons.18. The one or more non-transitory computer-readable media of claim 17,wherein the IWB appliance acquires the one or more facial images of, orone or more words spoken by, the particular person from the plurality ofpersons without involvement of the particular person.
 19. The one ormore non-transitory computer-readable media of claim 11, wherein the oneor more notifications pertaining to the particular electronic meetinginclude one or more of an announcement of the meeting or a welcome tothe meeting.
 20. A computer-implemented method comprising: receiving, byan interactive whiteboard (IWB) appliance via a communicationsinterface, from one or more sensors, sensor data that indicates adetected presence of a person, identifying, based upon the sensor datathat indicates a detected presence of a person and identity informationthat specifies one or more of image data or voice data for each of aplurality of persons, a particular person from the plurality of personsthat corresponds to the sensor data that indicates a detected presenceof a person, determining, based upon meeting information and a currenttime, whether an electronic meeting involving the IWB appliance isscheduled at or around the current time, in response to determining,based upon the meeting information and the current time, that anelectronic meeting involving the IWB appliance is scheduled at or aroundthe current time, determining, based upon the meeting information,whether the particular person is a scheduled participant of theparticular electronic meeting involving the IWB appliance that isscheduled at or around the current time, in response to determining,based upon the meeting information, that the particular person is ascheduled participant of the particular electronic meeting involving theIWB appliance that is scheduled at or around the current time, generateand display on the display of the IWB appliance one or morenotifications pertaining to the particular electronic meeting.